[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250110 20:50:59.464418 416 test_util.cc:274] Using random seed: 1082345642
I20250110 20:50:59.475606 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:50:59.475689 416 ts_itest-base.cc:113] --------------
I20250110 20:50:59.475741 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:50:59.475796 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:50:59.475849 416 ts_itest-base.cc:116] --------------
2025-01-10T20:50:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:50:59Z Disabled control of system clock
I20250110 20:50:59.496251 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41711
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41711
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250110 20:50:59.810587 428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:50:59.810861 428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:50:59.810947 428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:50:59.818574 428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:50:59.818657 428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:50:59.818704 428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:50:59.818779 428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:50:59.827301 428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41711
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:50:59.828778 428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:50:59.830367 428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:50:59.840242 433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:50:59.840302 434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:50:59.840449 428 server_base.cc:1034] running on GCE node
W20250110 20:50:59.840255 436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:50:59.841334 428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:50:59.842533 428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:50:59.843727 428 hybrid_clock.cc:648] HybridClock initialized: now 1736542259843635 us; error 72 us; skew 500 ppm
I20250110 20:50:59.844002 428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:50:59.846459 428 webserver.cc:458] Webserver started at http://127.0.104.62:42181/ using document root <none> and password file <none>
I20250110 20:50:59.847137 428 fs_manager.cc:362] Metadata directory not provided
I20250110 20:50:59.847244 428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:50:59.847515 428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:50:59.849871 428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "6087ff14ca6144519bafcb9e7cd60358"
format_stamp: "Formatted at 2025-01-10 20:50:59 on dist-test-slave-1h8z"
I20250110 20:50:59.850570 428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "6087ff14ca6144519bafcb9e7cd60358"
format_stamp: "Formatted at 2025-01-10 20:50:59 on dist-test-slave-1h8z"
I20250110 20:50:59.854986 428 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20250110 20:50:59.857767 442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:50:59.858952 428 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:50:59.859081 428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "6087ff14ca6144519bafcb9e7cd60358"
format_stamp: "Formatted at 2025-01-10 20:50:59 on dist-test-slave-1h8z"
I20250110 20:50:59.859239 428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:50:59.874398 428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:50:59.875152 428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:50:59.875402 428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:50:59.891024 428 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41711
I20250110 20:50:59.891054 493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41711 every 8 connection(s)
I20250110 20:50:59.892222 428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20250110 20:50:59.895972 494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:50:59.900058 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 428
I20250110 20:50:59.900321 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20250110 20:50:59.906181 494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358: Bootstrap starting.
I20250110 20:50:59.909632 494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358: Neither blocks nor log segments found. Creating new log.
I20250110 20:50:59.910555 494 log.cc:826] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358: Log is configured to *not* fsync() on all Append() calls
I20250110 20:50:59.913290 494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358: No bootstrap required, opened a new log
I20250110 20:50:59.921439 494 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6087ff14ca6144519bafcb9e7cd60358" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41711 } }
I20250110 20:50:59.921830 494 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:50:59.921926 494 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6087ff14ca6144519bafcb9e7cd60358, State: Initialized, Role: FOLLOWER
I20250110 20:50:59.922600 494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6087ff14ca6144519bafcb9e7cd60358" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41711 } }
I20250110 20:50:59.922869 494 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:50:59.923013 494 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:50:59.923187 494 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:50:59.925633 494 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6087ff14ca6144519bafcb9e7cd60358" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41711 } }
I20250110 20:50:59.926282 494 leader_election.cc:304] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6087ff14ca6144519bafcb9e7cd60358; no voters:
I20250110 20:50:59.926968 494 leader_election.cc:290] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:50:59.927104 499 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:50:59.928997 494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:50:59.929104 499 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [term 1 LEADER]: Becoming Leader. State: Replica: 6087ff14ca6144519bafcb9e7cd60358, State: Running, Role: LEADER
I20250110 20:50:59.929628 499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6087ff14ca6144519bafcb9e7cd60358" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41711 } }
I20250110 20:50:59.932511 500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6087ff14ca6144519bafcb9e7cd60358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6087ff14ca6144519bafcb9e7cd60358" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41711 } } }
I20250110 20:50:59.932829 500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [sys.catalog]: This master's current role is: LEADER
I20250110 20:50:59.932806 501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6087ff14ca6144519bafcb9e7cd60358. Latest consensus state: current_term: 1 leader_uuid: "6087ff14ca6144519bafcb9e7cd60358" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6087ff14ca6144519bafcb9e7cd60358" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41711 } } }
I20250110 20:50:59.933059 501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358 [sys.catalog]: This master's current role is: LEADER
I20250110 20:50:59.936842 509 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:50:59.940917 509 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:50:59.948937 509 catalog_manager.cc:1348] Generated new cluster ID: 31d0ee85f97e46d8a086572c478c79d1
I20250110 20:50:59.949033 509 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:50:59.968559 509 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:50:59.969785 509 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:50:59.978960 509 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6087ff14ca6144519bafcb9e7cd60358: Generated new TSK 0
I20250110 20:50:59.979732 509 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:50:59.986506 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--builtin_ntp_servers=127.0.104.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20250110 20:51:00.309466 518 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:51:00.309744 518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:00.309827 518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:00.309928 518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:00.317615 518 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250110 20:51:00.317713 518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:00.317888 518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:51:00.326462 518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:00.327891 518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:00.329907 518 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:00.339802 523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:00.339876 524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:00.339879 526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:00.340716 518 server_base.cc:1034] running on GCE node
I20250110 20:51:00.341214 518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:00.342010 518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:00.343204 518 hybrid_clock.cc:648] HybridClock initialized: now 1736542260343150 us; error 31 us; skew 500 ppm
I20250110 20:51:00.343469 518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:00.350946 518 webserver.cc:458] Webserver started at http://127.0.104.1:40217/ using document root <none> and password file <none>
I20250110 20:51:00.351657 518 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:00.352162 518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:00.352452 518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:00.354875 518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "346a0a9d4e234f9e85e24d295d41ed37"
format_stamp: "Formatted at 2025-01-10 20:51:00 on dist-test-slave-1h8z"
I20250110 20:51:00.355559 518 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "346a0a9d4e234f9e85e24d295d41ed37"
format_stamp: "Formatted at 2025-01-10 20:51:00 on dist-test-slave-1h8z"
I20250110 20:51:00.360147 518 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250110 20:51:00.363075 532 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:00.364251 518 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:51:00.364382 518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "346a0a9d4e234f9e85e24d295d41ed37"
format_stamp: "Formatted at 2025-01-10 20:51:00 on dist-test-slave-1h8z"
I20250110 20:51:00.364547 518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:00.380402 518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:00.381058 518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:00.381304 518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:00.382074 518 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:00.383414 518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:00.383484 518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:00.383589 518 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:00.383642 518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:00.412703 518 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35691
I20250110 20:51:00.412734 644 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35691 every 8 connection(s)
I20250110 20:51:00.413851 518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20250110 20:51:00.421463 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 518
I20250110 20:51:00.421667 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20250110 20:51:00.424885 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--builtin_ntp_servers=127.0.104.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20250110 20:51:00.426348 645 heartbeater.cc:346] Connected to a master server at 127.0.104.62:41711
I20250110 20:51:00.426651 645 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:00.427579 645 heartbeater.cc:510] Master 127.0.104.62:41711 requested a full tablet report, sending...
I20250110 20:51:00.429750 459 ts_manager.cc:194] Registered new tserver with Master: 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691)
I20250110 20:51:00.431259 459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:54661
W20250110 20:51:00.736810 648 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:51:00.737102 648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:00.737179 648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:00.737280 648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:00.744570 648 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250110 20:51:00.744686 648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:00.744863 648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:51:00.753986 648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:00.755488 648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:00.757091 648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:00.767426 653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:00.767494 656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:00.767549 654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:00.768021 648 server_base.cc:1034] running on GCE node
I20250110 20:51:00.768584 648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:00.769438 648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:00.770637 648 hybrid_clock.cc:648] HybridClock initialized: now 1736542260770583 us; error 33 us; skew 500 ppm
I20250110 20:51:00.770900 648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:00.772804 648 webserver.cc:458] Webserver started at http://127.0.104.2:45171/ using document root <none> and password file <none>
I20250110 20:51:00.773417 648 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:00.773509 648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:00.773751 648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:00.776067 648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "c1d32e2dc47b4546a3e87001158d23ad"
format_stamp: "Formatted at 2025-01-10 20:51:00 on dist-test-slave-1h8z"
I20250110 20:51:00.776708 648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "c1d32e2dc47b4546a3e87001158d23ad"
format_stamp: "Formatted at 2025-01-10 20:51:00 on dist-test-slave-1h8z"
I20250110 20:51:00.781674 648 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20250110 20:51:00.784512 662 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:00.785642 648 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250110 20:51:00.785768 648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "c1d32e2dc47b4546a3e87001158d23ad"
format_stamp: "Formatted at 2025-01-10 20:51:00 on dist-test-slave-1h8z"
I20250110 20:51:00.785917 648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:00.799787 648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:00.800467 648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:00.800706 648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:00.801502 648 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:00.803280 648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:00.803347 648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:00.803452 648 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:00.803498 648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:00.834614 648 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44429
I20250110 20:51:00.834635 774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44429 every 8 connection(s)
I20250110 20:51:00.835884 648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20250110 20:51:00.840032 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 648
I20250110 20:51:00.840158 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20250110 20:51:00.843257 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--builtin_ntp_servers=127.0.104.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20250110 20:51:00.848723 775 heartbeater.cc:346] Connected to a master server at 127.0.104.62:41711
I20250110 20:51:00.848990 775 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:00.849711 775 heartbeater.cc:510] Master 127.0.104.62:41711 requested a full tablet report, sending...
I20250110 20:51:00.851015 459 ts_manager.cc:194] Registered new tserver with Master: c1d32e2dc47b4546a3e87001158d23ad (127.0.104.2:44429)
I20250110 20:51:00.851732 459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:37971
W20250110 20:51:01.163523 778 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:51:01.163825 778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:01.163894 778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:01.163966 778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:01.171692 778 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250110 20:51:01.171792 778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:01.171953 778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:51:01.181115 778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:01.182556 778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:01.184154 778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:01.194890 784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:01.194859 786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:01.195051 783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:01.195796 778 server_base.cc:1034] running on GCE node
I20250110 20:51:01.196280 778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:01.197136 778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:01.198328 778 hybrid_clock.cc:648] HybridClock initialized: now 1736542261198268 us; error 40 us; skew 500 ppm
I20250110 20:51:01.198575 778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:01.200562 778 webserver.cc:458] Webserver started at http://127.0.104.3:34955/ using document root <none> and password file <none>
I20250110 20:51:01.201200 778 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:01.201288 778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:01.201529 778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:01.203934 778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "a8a61641151b4b7493b57e75e0b0f04a"
format_stamp: "Formatted at 2025-01-10 20:51:01 on dist-test-slave-1h8z"
I20250110 20:51:01.204574 778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "a8a61641151b4b7493b57e75e0b0f04a"
format_stamp: "Formatted at 2025-01-10 20:51:01 on dist-test-slave-1h8z"
I20250110 20:51:01.209131 778 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250110 20:51:01.212038 792 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:01.213184 778 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250110 20:51:01.213301 778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "a8a61641151b4b7493b57e75e0b0f04a"
format_stamp: "Formatted at 2025-01-10 20:51:01 on dist-test-slave-1h8z"
I20250110 20:51:01.213440 778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:01.231070 778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:01.231779 778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:01.232002 778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:01.232813 778 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:01.234177 778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:01.234243 778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:01.234328 778 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:01.234366 778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:01.266153 778 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42401
I20250110 20:51:01.266182 904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42401 every 8 connection(s)
I20250110 20:51:01.267414 778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20250110 20:51:01.267838 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 778
I20250110 20:51:01.268034 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20250110 20:51:01.279649 905 heartbeater.cc:346] Connected to a master server at 127.0.104.62:41711
I20250110 20:51:01.279932 905 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:01.280717 905 heartbeater.cc:510] Master 127.0.104.62:41711 requested a full tablet report, sending...
I20250110 20:51:01.282037 459 ts_manager.cc:194] Registered new tserver with Master: a8a61641151b4b7493b57e75e0b0f04a (127.0.104.3:42401)
I20250110 20:51:01.282836 459 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:50287
I20250110 20:51:01.284492 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:51:01.288095 416 test_util.cc:274] Using random seed: 1084169339
I20250110 20:51:01.309043 459 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33220:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:51:01.310431 459 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:51:01.346966 710 tablet_service.cc:1467] Processing CreateTablet for tablet ae7f40342b3e4efca7d655f22362bd09 (DEFAULT_TABLE table=test-workload [id=d0b7a4c185db4915ae2702c9a26a3d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:01.347242 840 tablet_service.cc:1467] Processing CreateTablet for tablet ae7f40342b3e4efca7d655f22362bd09 (DEFAULT_TABLE table=test-workload [id=d0b7a4c185db4915ae2702c9a26a3d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:01.347883 580 tablet_service.cc:1467] Processing CreateTablet for tablet ae7f40342b3e4efca7d655f22362bd09 (DEFAULT_TABLE table=test-workload [id=d0b7a4c185db4915ae2702c9a26a3d5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:01.348472 710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7f40342b3e4efca7d655f22362bd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:01.348489 840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7f40342b3e4efca7d655f22362bd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:01.348999 580 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7f40342b3e4efca7d655f22362bd09. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:01.359575 923 tablet_bootstrap.cc:492] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: Bootstrap starting.
I20250110 20:51:01.360819 924 tablet_bootstrap.cc:492] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad: Bootstrap starting.
I20250110 20:51:01.361654 925 tablet_bootstrap.cc:492] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: Bootstrap starting.
I20250110 20:51:01.362967 923 tablet_bootstrap.cc:654] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:01.364197 923 log.cc:826] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:01.364502 925 tablet_bootstrap.cc:654] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:01.364609 924 tablet_bootstrap.cc:654] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:01.365348 925 log.cc:826] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:01.365741 924 log.cc:826] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:01.367652 925 tablet_bootstrap.cc:492] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: No bootstrap required, opened a new log
I20250110 20:51:01.367859 925 ts_tablet_manager.cc:1397] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20250110 20:51:01.367978 924 tablet_bootstrap.cc:492] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad: No bootstrap required, opened a new log
I20250110 20:51:01.368194 924 ts_tablet_manager.cc:1397] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20250110 20:51:01.369418 923 tablet_bootstrap.cc:492] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: No bootstrap required, opened a new log
I20250110 20:51:01.369644 923 ts_tablet_manager.cc:1397] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20250110 20:51:01.374634 924 raft_consensus.cc:357] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.374974 924 raft_consensus.cc:738] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d32e2dc47b4546a3e87001158d23ad, State: Initialized, Role: FOLLOWER
I20250110 20:51:01.375810 924 consensus_queue.cc:260] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.376102 925 raft_consensus.cc:357] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.376467 925 raft_consensus.cc:738] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8a61641151b4b7493b57e75e0b0f04a, State: Initialized, Role: FOLLOWER
I20250110 20:51:01.376843 775 heartbeater.cc:502] Master 127.0.104.62:41711 was elected leader, sending a full tablet report...
I20250110 20:51:01.376904 925 consensus_queue.cc:260] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.377314 924 ts_tablet_manager.cc:1428] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20250110 20:51:01.377664 923 raft_consensus.cc:357] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.378047 923 raft_consensus.cc:738] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 346a0a9d4e234f9e85e24d295d41ed37, State: Initialized, Role: FOLLOWER
I20250110 20:51:01.378489 923 consensus_queue.cc:260] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.379490 645 heartbeater.cc:502] Master 127.0.104.62:41711 was elected leader, sending a full tablet report...
I20250110 20:51:01.380038 923 ts_tablet_manager.cc:1428] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20250110 20:51:01.380995 905 heartbeater.cc:502] Master 127.0.104.62:41711 was elected leader, sending a full tablet report...
I20250110 20:51:01.381469 925 ts_tablet_manager.cc:1428] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: Time spent starting tablet: real 0.013s user 0.005s sys 0.007s
W20250110 20:51:01.410394 640 generic_service.cc:139] {username='slave'} at 127.0.0.1:50684 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20250110 20:51:01.410571 640 generic_service.cc:175] {username='slave'} at 127.0.0.1:50684 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.0.104.1:35691'
W20250110 20:51:01.411784 640 generic_service.cc:139] {username='slave'} at 127.0.0.1:50684 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:01.411863 640 generic_service.cc:175] {username='slave'} at 127.0.0.1:50684 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250110 20:51:01.415817 646 tablet.cc:2367] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:51:01.417629 770 generic_service.cc:139] {username='slave'} at 127.0.0.1:56696 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20250110 20:51:01.417814 770 generic_service.cc:175] {username='slave'} at 127.0.0.1:56696 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.0.104.1:35691'
W20250110 20:51:01.418864 770 generic_service.cc:139] {username='slave'} at 127.0.0.1:56696 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:01.418951 770 generic_service.cc:175] {username='slave'} at 127.0.0.1:56696 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250110 20:51:01.424161 900 generic_service.cc:139] {username='slave'} at 127.0.0.1:43434 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20250110 20:51:01.424347 900 generic_service.cc:175] {username='slave'} at 127.0.0.1:43434 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.0.104.1:35691'
W20250110 20:51:01.425336 900 generic_service.cc:139] {username='slave'} at 127.0.0.1:43434 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:01.425424 900 generic_service.cc:175] {username='slave'} at 127.0.0.1:43434 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20250110 20:51:01.426505 730 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae7f40342b3e4efca7d655f22362bd09"
dest_uuid: "c1d32e2dc47b4546a3e87001158d23ad"
from {username='slave'} at 127.0.0.1:56690
I20250110 20:51:01.426820 730 raft_consensus.cc:491] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:51:01.426987 730 raft_consensus.cc:3054] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:01.428961 730 raft_consensus.cc:513] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.431157 730 leader_election.cc:290] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [CANDIDATE]: Term 1 election: Requested vote from peers 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691), a8a61641151b4b7493b57e75e0b0f04a (127.0.104.3:42401)
W20250110 20:51:01.431563 934 leader_election.cc:336] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691
I20250110 20:51:01.437566 860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae7f40342b3e4efca7d655f22362bd09" candidate_uuid: "c1d32e2dc47b4546a3e87001158d23ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a8a61641151b4b7493b57e75e0b0f04a"
I20250110 20:51:01.437896 860 raft_consensus.cc:3054] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:01.440531 860 raft_consensus.cc:2463] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1d32e2dc47b4546a3e87001158d23ad in term 1.
I20250110 20:51:01.440982 664 leader_election.cc:304] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a8a61641151b4b7493b57e75e0b0f04a, c1d32e2dc47b4546a3e87001158d23ad; no voters: 346a0a9d4e234f9e85e24d295d41ed37
I20250110 20:51:01.441377 929 raft_consensus.cc:2798] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:51:01.441803 929 raft_consensus.cc:695] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 1 LEADER]: Becoming Leader. State: Replica: c1d32e2dc47b4546a3e87001158d23ad, State: Running, Role: LEADER
I20250110 20:51:01.442221 929 consensus_queue.cc:237] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:01.446975 456 catalog_manager.cc:5526] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad reported cstate change: term changed from 0 to 1, leader changed from <none> to c1d32e2dc47b4546a3e87001158d23ad (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c1d32e2dc47b4546a3e87001158d23ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } health_report { overall_health: HEALTHY } } }
W20250110 20:51:01.519030 906 tablet.cc:2367] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:51:01.560505 929 consensus_queue.cc:1035] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:51:01.566572 934 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Couldn't send request to peer 346a0a9d4e234f9e85e24d295d41ed37. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:51:02.049250 934 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Couldn't send request to peer 346a0a9d4e234f9e85e24d295d41ed37. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691. This is attempt 6: this message will repeat every 5th retry.
W20250110 20:51:02.552207 934 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Couldn't send request to peer 346a0a9d4e234f9e85e24d295d41ed37. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691. This is attempt 11: this message will repeat every 5th retry.
W20250110 20:51:03.079474 934 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Couldn't send request to peer 346a0a9d4e234f9e85e24d295d41ed37. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691. This is attempt 16: this message will repeat every 5th retry.
W20250110 20:51:03.566247 934 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Couldn't send request to peer 346a0a9d4e234f9e85e24d295d41ed37. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691. This is attempt 21: this message will repeat every 5th retry.
W20250110 20:51:04.067601 934 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer 346a0a9d4e234f9e85e24d295d41ed37 (127.0.104.1:35691): Couldn't send request to peer 346a0a9d4e234f9e85e24d295d41ed37. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.0.104.1:35691. This is attempt 26: this message will repeat every 5th retry.
I20250110 20:51:04.475912 942 consensus_queue.cc:579] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Leader has been unable to successfully communicate with peer 346a0a9d4e234f9e85e24d295d41ed37 for more than 3 seconds (3.033s)
W20250110 20:51:04.478925 458 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae7f40342b3e4efca7d655f22362bd09 with cas_config_opid_index -1: no extra replica candidate found for tablet ae7f40342b3e4efca7d655f22362bd09 (table test-workload [id=d0b7a4c185db4915ae2702c9a26a3d5c]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250110 20:51:04.515295 640 generic_service.cc:139] {username='slave'} at 127.0.0.1:50684 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:04.515416 640 generic_service.cc:175] {username='slave'} at 127.0.0.1:50684 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20250110 20:51:04.516345 770 generic_service.cc:139] {username='slave'} at 127.0.0.1:56696 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:04.516443 770 generic_service.cc:175] {username='slave'} at 127.0.0.1:56696 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20250110 20:51:04.517316 900 generic_service.cc:139] {username='slave'} at 127.0.0.1:43434 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:04.517423 900 generic_service.cc:175] {username='slave'} at 127.0.0.1:43434 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20250110 20:51:04.561065 600 raft_consensus.cc:3054] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:04.679342 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.4:0
--local_ip_for_outbound_sockets=127.0.104.4
--webserver_interface=127.0.104.4
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--builtin_ntp_servers=127.0.104.20:36329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20250110 20:51:04.999814 950 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:51:05.000104 950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:05.000205 950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:05.000312 950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:05.009810 950 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250110 20:51:05.009930 950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:05.010113 950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.4
I20250110 20:51:05.019367 950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.104.4
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.0.104.4
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:05.020828 950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:05.022403 950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:05.033206 955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:05.033274 958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:05.033243 956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:05.034114 950 server_base.cc:1034] running on GCE node
I20250110 20:51:05.034639 950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:05.035501 950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:05.036727 950 hybrid_clock.cc:648] HybridClock initialized: now 1736542265036663 us; error 36 us; skew 500 ppm
I20250110 20:51:05.037004 950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:05.038992 950 webserver.cc:458] Webserver started at http://127.0.104.4:44637/ using document root <none> and password file <none>
I20250110 20:51:05.039644 950 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:05.039757 950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:05.040023 950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:05.042927 950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "f5978e5fd0084d7f91be003d147d640f"
format_stamp: "Formatted at 2025-01-10 20:51:05 on dist-test-slave-1h8z"
I20250110 20:51:05.043619 950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "f5978e5fd0084d7f91be003d147d640f"
format_stamp: "Formatted at 2025-01-10 20:51:05 on dist-test-slave-1h8z"
I20250110 20:51:05.048420 950 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250110 20:51:05.051402 964 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:05.052547 950 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:51:05.052681 950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "f5978e5fd0084d7f91be003d147d640f"
format_stamp: "Formatted at 2025-01-10 20:51:05 on dist-test-slave-1h8z"
I20250110 20:51:05.052845 950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:05.082612 950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:05.083398 950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:05.083654 950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:05.084527 950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:05.085965 950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:05.086043 950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:05.086156 950 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:05.086236 950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:05.117064 950 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.4:40065
I20250110 20:51:05.117100 1076 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.4:40065 every 8 connection(s)
I20250110 20:51:05.118317 950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20250110 20:51:05.128553 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 950
I20250110 20:51:05.128733 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20250110 20:51:05.131083 1077 heartbeater.cc:346] Connected to a master server at 127.0.104.62:41711
I20250110 20:51:05.131397 1077 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:05.132231 1077 heartbeater.cc:510] Master 127.0.104.62:41711 requested a full tablet report, sending...
I20250110 20:51:05.133450 456 ts_manager.cc:194] Registered new tserver with Master: f5978e5fd0084d7f91be003d147d640f (127.0.104.4:40065)
I20250110 20:51:05.134317 456 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.4:59109
I20250110 20:51:05.144965 730 consensus_queue.cc:237] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } peers { permanent_uuid: "f5978e5fd0084d7f91be003d147d640f" member_type: NON_VOTER last_known_addr { host: "127.0.104.4" port: 40065 } attrs { promote: true } }
I20250110 20:51:05.147706 599 raft_consensus.cc:1270] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 1 FOLLOWER]: Refusing update from remote peer c1d32e2dc47b4546a3e87001158d23ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:51:05.147993 860 raft_consensus.cc:1270] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 1 FOLLOWER]: Refusing update from remote peer c1d32e2dc47b4546a3e87001158d23ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:51:05.148418 942 consensus_queue.cc:1035] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:51:05.148751 943 consensus_queue.cc:1035] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:51:05.151654 944 raft_consensus.cc:2949] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } peers { permanent_uuid: "f5978e5fd0084d7f91be003d147d640f" member_type: NON_VOTER last_known_addr { host: "127.0.104.4" port: 40065 } attrs { promote: true } } }
I20250110 20:51:05.152279 599 raft_consensus.cc:2949] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } peers { permanent_uuid: "f5978e5fd0084d7f91be003d147d640f" member_type: NON_VOTER last_known_addr { host: "127.0.104.4" port: 40065 } attrs { promote: true } } }
I20250110 20:51:05.154408 860 raft_consensus.cc:2949] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } peers { permanent_uuid: "f5978e5fd0084d7f91be003d147d640f" member_type: NON_VOTER last_known_addr { host: "127.0.104.4" port: 40065 } attrs { promote: true } } }
I20250110 20:51:05.155769 446 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ae7f40342b3e4efca7d655f22362bd09 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
W20250110 20:51:05.157984 664 consensus_peers.cc:487] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad -> Peer f5978e5fd0084d7f91be003d147d640f (127.0.104.4:40065): Couldn't send request to peer f5978e5fd0084d7f91be003d147d640f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae7f40342b3e4efca7d655f22362bd09. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:51:05.159848 456 catalog_manager.cc:5526] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad reported cstate change: config changed from index -1 to 2, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) added. New cstate: current_term: 1 leader_uuid: "c1d32e2dc47b4546a3e87001158d23ad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5978e5fd0084d7f91be003d147d640f" member_type: NON_VOTER last_known_addr { host: "127.0.104.4" port: 40065 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250110 20:51:05.165343 730 consensus_queue.cc:237] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } }
I20250110 20:51:05.166816 599 raft_consensus.cc:1270] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 1 FOLLOWER]: Refusing update from remote peer c1d32e2dc47b4546a3e87001158d23ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 20:51:05.167006 860 raft_consensus.cc:1270] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 1 FOLLOWER]: Refusing update from remote peer c1d32e2dc47b4546a3e87001158d23ad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 20:51:05.167351 942 consensus_queue.cc:1035] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 20:51:05.167579 944 consensus_queue.cc:1035] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 20:51:05.169366 942 raft_consensus.cc:2949] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } }
I20250110 20:51:05.169912 599 raft_consensus.cc:2949] T ae7f40342b3e4efca7d655f22362bd09 P 346a0a9d4e234f9e85e24d295d41ed37 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } }
I20250110 20:51:05.172114 446 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet ae7f40342b3e4efca7d655f22362bd09 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250110 20:51:05.172960 860 raft_consensus.cc:2949] T ae7f40342b3e4efca7d655f22362bd09 P a8a61641151b4b7493b57e75e0b0f04a [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } } }
I20250110 20:51:05.173063 459 catalog_manager.cc:5526] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad reported cstate change: config changed from index 2 to 3, NON_VOTER f5978e5fd0084d7f91be003d147d640f (127.0.104.4) evicted. New cstate: current_term: 1 leader_uuid: "c1d32e2dc47b4546a3e87001158d23ad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "346a0a9d4e234f9e85e24d295d41ed37" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8a61641151b4b7493b57e75e0b0f04a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d32e2dc47b4546a3e87001158d23ad" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44429 } health_report { overall_health: HEALTHY } } }
W20250110 20:51:05.183444 640 generic_service.cc:139] {username='slave'} at 127.0.0.1:50684 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:05.183543 640 generic_service.cc:175] {username='slave'} at 127.0.0.1:50684 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20250110 20:51:05.182837 1012 tablet_service.cc:1514] Processing DeleteTablet for tablet ae7f40342b3e4efca7d655f22362bd09 with delete_type TABLET_DATA_TOMBSTONED (TS f5978e5fd0084d7f91be003d147d640f not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:55618
W20250110 20:51:05.184566 770 generic_service.cc:139] {username='slave'} at 127.0.0.1:56696 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:05.184650 770 generic_service.cc:175] {username='slave'} at 127.0.0.1:56696 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250110 20:51:05.185295 444 catalog_manager.cc:4839] TS f5978e5fd0084d7f91be003d147d640f (127.0.104.4:40065): delete failed for tablet ae7f40342b3e4efca7d655f22362bd09 because the tablet was not found. No further retry: Not found: Tablet not found: ae7f40342b3e4efca7d655f22362bd09
W20250110 20:51:05.185557 900 generic_service.cc:139] {username='slave'} at 127.0.0.1:43434 forcing change of non-runtime-safe flag fail_dns_resolution
I20250110 20:51:05.185644 900 generic_service.cc:175] {username='slave'} at 127.0.0.1:43434 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20250110 20:51:05.186609 730 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae7f40342b3e4efca7d655f22362bd09"
dest_uuid: "c1d32e2dc47b4546a3e87001158d23ad"
mode: GRACEFUL
new_leader_uuid: "c1d32e2dc47b4546a3e87001158d23ad"
from {username='slave'} at 127.0.0.1:56690
I20250110 20:51:05.186837 730 raft_consensus.cc:604] T ae7f40342b3e4efca7d655f22362bd09 P c1d32e2dc47b4546a3e87001158d23ad [term 1 LEADER]: Received request to transfer leadership to TS c1d32e2dc47b4546a3e87001158d23ad
I20250110 20:51:06.136408 1077 heartbeater.cc:502] Master 127.0.104.62:41711 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/raft_consensus_election-itest.cc:291: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
I20250110 20:51:35.270102 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 518
I20250110 20:51:35.271605 640 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:35.364873 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 518
I20250110 20:51:35.379451 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 648
I20250110 20:51:35.380312 770 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:35.473152 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 648
I20250110 20:51:35.493537 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 778
I20250110 20:51:35.494432 900 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:35.592299 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 778
I20250110 20:51:35.607061 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 950
I20250110 20:51:35.612679 1072 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:35.693094 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 950
I20250110 20:51:35.706120 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 428
I20250110 20:51:35.707183 489 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:35.810057 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 428
2025-01-10T20:51:35Z chronyd exiting
I20250110 20:51:35.839859 416 test_util.cc:181] -----------------------------------------------
I20250110 20:51:35.839937 416 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736542259391478-416-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (36379 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20250110 20:51:35.843180 416 test_util.cc:274] Using random seed: 1118724429
I20250110 20:51:35.844812 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:51:35.844879 416 ts_itest-base.cc:113] --------------
I20250110 20:51:35.844928 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:51:35.844982 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:51:35.845029 416 ts_itest-base.cc:116] --------------
2025-01-10T20:51:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:51:35Z Disabled control of system clock
I20250110 20:51:35.863677 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33135
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33135 with env {}
W20250110 20:51:36.161804 1114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:36.162088 1114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:36.162218 1114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:36.169713 1114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:51:36.169827 1114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:36.169893 1114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:51:36.169965 1114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:51:36.179011 1114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33135
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:36.180555 1114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:36.182178 1114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:36.192030 1120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:36.192006 1122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:36.192239 1119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:36.192692 1114 server_base.cc:1034] running on GCE node
I20250110 20:51:36.193342 1114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:36.194551 1114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:36.195747 1114 hybrid_clock.cc:648] HybridClock initialized: now 1736542296195688 us; error 35 us; skew 500 ppm
I20250110 20:51:36.196051 1114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:36.198045 1114 webserver.cc:458] Webserver started at http://127.0.104.62:40667/ using document root <none> and password file <none>
I20250110 20:51:36.198700 1114 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:36.198846 1114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:36.199142 1114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:36.201530 1114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "6d626cafb8cf4ed5813a7c8afb66b34d"
format_stamp: "Formatted at 2025-01-10 20:51:36 on dist-test-slave-1h8z"
I20250110 20:51:36.202234 1114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "6d626cafb8cf4ed5813a7c8afb66b34d"
format_stamp: "Formatted at 2025-01-10 20:51:36 on dist-test-slave-1h8z"
I20250110 20:51:36.206853 1114 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20250110 20:51:36.209717 1128 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:36.210920 1114 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20250110 20:51:36.211052 1114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "6d626cafb8cf4ed5813a7c8afb66b34d"
format_stamp: "Formatted at 2025-01-10 20:51:36 on dist-test-slave-1h8z"
I20250110 20:51:36.211201 1114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:36.236392 1114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:36.237109 1114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:36.237351 1114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:36.252828 1114 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33135
I20250110 20:51:36.252861 1179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33135 every 8 connection(s)
I20250110 20:51:36.254025 1114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:51:36.255846 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1114
I20250110 20:51:36.256026 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:51:36.257560 1180 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:36.269830 1180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d: Bootstrap starting.
I20250110 20:51:36.272650 1180 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:36.273723 1180 log.cc:826] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:36.275985 1180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d: No bootstrap required, opened a new log
I20250110 20:51:36.281738 1180 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33135 } }
I20250110 20:51:36.282030 1180 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:36.282100 1180 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d626cafb8cf4ed5813a7c8afb66b34d, State: Initialized, Role: FOLLOWER
I20250110 20:51:36.282580 1180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33135 } }
I20250110 20:51:36.282784 1180 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:51:36.282891 1180 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:51:36.283020 1180 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:36.285096 1180 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33135 } }
I20250110 20:51:36.285501 1180 leader_election.cc:304] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d626cafb8cf4ed5813a7c8afb66b34d; no voters:
I20250110 20:51:36.285997 1180 leader_election.cc:290] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:51:36.286115 1185 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:51:36.286705 1185 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [term 1 LEADER]: Becoming Leader. State: Replica: 6d626cafb8cf4ed5813a7c8afb66b34d, State: Running, Role: LEADER
I20250110 20:51:36.287294 1185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33135 } }
I20250110 20:51:36.287791 1180 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:51:36.289898 1187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d626cafb8cf4ed5813a7c8afb66b34d. Latest consensus state: current_term: 1 leader_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33135 } } }
I20250110 20:51:36.290217 1187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [sys.catalog]: This master's current role is: LEADER
I20250110 20:51:36.294971 1186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d626cafb8cf4ed5813a7c8afb66b34d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33135 } } }
I20250110 20:51:36.295325 1186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d [sys.catalog]: This master's current role is: LEADER
I20250110 20:51:36.296092 1192 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:51:36.300416 1192 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:51:36.307454 1192 catalog_manager.cc:1348] Generated new cluster ID: 965799c416e24d53a9e3b2ea5f01d256
I20250110 20:51:36.307561 1192 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:51:36.336117 1192 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:51:36.337769 1192 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:51:36.346835 1192 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 6d626cafb8cf4ed5813a7c8afb66b34d: Generated new TSK 0
I20250110 20:51:36.347730 1192 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:51:36.362619 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33135
--builtin_ntp_servers=127.0.104.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20250110 20:51:36.662847 1204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:36.663098 1204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:36.663220 1204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:36.670444 1204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:36.670651 1204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:51:36.679278 1204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:36.680749 1204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:36.682380 1204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:36.693562 1210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:36.693595 1209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:36.693634 1212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:36.694671 1204 server_base.cc:1034] running on GCE node
I20250110 20:51:36.695226 1204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:36.696048 1204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:36.697243 1204 hybrid_clock.cc:648] HybridClock initialized: now 1736542296697190 us; error 37 us; skew 500 ppm
I20250110 20:51:36.697495 1204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:36.699530 1204 webserver.cc:458] Webserver started at http://127.0.104.1:37851/ using document root <none> and password file <none>
I20250110 20:51:36.700186 1204 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:36.700292 1204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:36.700565 1204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:36.702950 1204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bbb9c58f778d4c65a3660ac7479651ba"
format_stamp: "Formatted at 2025-01-10 20:51:36 on dist-test-slave-1h8z"
I20250110 20:51:36.703637 1204 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bbb9c58f778d4c65a3660ac7479651ba"
format_stamp: "Formatted at 2025-01-10 20:51:36 on dist-test-slave-1h8z"
I20250110 20:51:36.708153 1204 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250110 20:51:36.711050 1218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:36.712226 1204 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:51:36.712364 1204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bbb9c58f778d4c65a3660ac7479651ba"
format_stamp: "Formatted at 2025-01-10 20:51:36 on dist-test-slave-1h8z"
I20250110 20:51:36.712513 1204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:36.742338 1204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:36.743166 1204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:36.743422 1204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:36.744360 1204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:36.745810 1204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:36.745883 1204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:36.746006 1204 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:36.746058 1204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:36.773633 1204 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39637
I20250110 20:51:36.773685 1330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39637 every 8 connection(s)
I20250110 20:51:36.774890 1204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:51:36.776160 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1204
I20250110 20:51:36.776429 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:51:36.779873 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33135
--builtin_ntp_servers=127.0.104.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20250110 20:51:36.790052 1331 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33135
I20250110 20:51:36.790333 1331 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:36.791144 1331 heartbeater.cc:510] Master 127.0.104.62:33135 requested a full tablet report, sending...
I20250110 20:51:36.793355 1145 ts_manager.cc:194] Registered new tserver with Master: bbb9c58f778d4c65a3660ac7479651ba (127.0.104.1:39637)
I20250110 20:51:36.795454 1145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:39503
W20250110 20:51:37.081544 1335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:37.081801 1335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:37.081881 1335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:37.089114 1335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:37.089289 1335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:51:37.098111 1335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:37.099668 1335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:37.101233 1335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:37.111560 1343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:37.111588 1340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:37.111616 1341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:37.112321 1335 server_base.cc:1034] running on GCE node
I20250110 20:51:37.112812 1335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:37.113627 1335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:37.114831 1335 hybrid_clock.cc:648] HybridClock initialized: now 1736542297114774 us; error 33 us; skew 500 ppm
I20250110 20:51:37.115089 1335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:37.117128 1335 webserver.cc:458] Webserver started at http://127.0.104.2:41105/ using document root <none> and password file <none>
I20250110 20:51:37.117796 1335 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:37.117904 1335 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:37.118170 1335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:37.120517 1335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:37.121191 1335 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:37.125692 1335 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250110 20:51:37.128563 1349 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:37.129699 1335 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250110 20:51:37.129835 1335 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:37.129977 1335 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:37.150380 1335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:37.151114 1335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:37.151355 1335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:37.152248 1335 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:37.153595 1335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:37.153668 1335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:37.153779 1335 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:37.153828 1335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:37.181510 1335 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43219
I20250110 20:51:37.181540 1461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43219 every 8 connection(s)
I20250110 20:51:37.182673 1335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:51:37.184190 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1335
I20250110 20:51:37.184376 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:51:37.187345 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33135
--builtin_ntp_servers=127.0.104.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20250110 20:51:37.195333 1462 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33135
I20250110 20:51:37.195578 1462 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:37.196309 1462 heartbeater.cc:510] Master 127.0.104.62:33135 requested a full tablet report, sending...
I20250110 20:51:37.197782 1145 ts_manager.cc:194] Registered new tserver with Master: e74bc04852a54c5db0d7d7f5d3fe5e25 (127.0.104.2:43219)
I20250110 20:51:37.198807 1145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:56515
W20250110 20:51:37.493659 1465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:37.493923 1465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:37.494009 1465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:37.501426 1465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:37.501602 1465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:51:37.510213 1465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:37.511682 1465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:37.513234 1465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:37.524118 1470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:37.524080 1473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:37.524145 1471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:37.525290 1465 server_base.cc:1034] running on GCE node
I20250110 20:51:37.525779 1465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:37.526541 1465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:37.527721 1465 hybrid_clock.cc:648] HybridClock initialized: now 1736542297527649 us; error 51 us; skew 500 ppm
I20250110 20:51:37.527992 1465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:37.529889 1465 webserver.cc:458] Webserver started at http://127.0.104.3:45083/ using document root <none> and password file <none>
I20250110 20:51:37.530526 1465 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:37.530630 1465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:37.530994 1465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:37.533330 1465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9021cdbe740d4eaf834b08c85ecce67d"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:37.533995 1465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9021cdbe740d4eaf834b08c85ecce67d"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:37.538508 1465 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250110 20:51:37.541478 1479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:37.542626 1465 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250110 20:51:37.542800 1465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9021cdbe740d4eaf834b08c85ecce67d"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:37.542960 1465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:37.583545 1465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:37.584308 1465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:37.584555 1465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:37.585415 1465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:37.586798 1465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:37.586872 1465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:37.586984 1465 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:37.587033 1465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:37.614979 1465 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45405
I20250110 20:51:37.615031 1591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45405 every 8 connection(s)
I20250110 20:51:37.616194 1465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:51:37.622462 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1465
I20250110 20:51:37.622632 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:51:37.627640 1592 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33135
I20250110 20:51:37.627874 1592 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:37.628803 1592 heartbeater.cc:510] Master 127.0.104.62:33135 requested a full tablet report, sending...
I20250110 20:51:37.630028 1145 ts_manager.cc:194] Registered new tserver with Master: 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405)
I20250110 20:51:37.630734 1145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:46005
I20250110 20:51:37.637401 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:51:37.650913 1145 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57154:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250110 20:51:37.656103 1145 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:51:37.692835 1527 tablet_service.cc:1467] Processing CreateTablet for tablet 3b422b3f077f43a6bef8b5a158fc60b9 (DEFAULT_TABLE table=TestTable [id=448a117c95ac417e8a5a6ecbc0727a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:37.692942 1397 tablet_service.cc:1467] Processing CreateTablet for tablet 3b422b3f077f43a6bef8b5a158fc60b9 (DEFAULT_TABLE table=TestTable [id=448a117c95ac417e8a5a6ecbc0727a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:37.693466 1266 tablet_service.cc:1467] Processing CreateTablet for tablet 3b422b3f077f43a6bef8b5a158fc60b9 (DEFAULT_TABLE table=TestTable [id=448a117c95ac417e8a5a6ecbc0727a50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:37.694075 1527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b422b3f077f43a6bef8b5a158fc60b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:37.694296 1397 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b422b3f077f43a6bef8b5a158fc60b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:37.694674 1266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b422b3f077f43a6bef8b5a158fc60b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:37.703706 1610 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Bootstrap starting.
I20250110 20:51:37.706732 1610 tablet_bootstrap.cc:654] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:37.706732 1611 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: Bootstrap starting.
I20250110 20:51:37.708063 1610 log.cc:826] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:37.708079 1612 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25: Bootstrap starting.
I20250110 20:51:37.709955 1611 tablet_bootstrap.cc:654] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:37.710868 1612 tablet_bootstrap.cc:654] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:37.711077 1611 log.cc:826] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:37.711797 1612 log.cc:826] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:37.711854 1610 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: No bootstrap required, opened a new log
I20250110 20:51:37.712083 1610 ts_tablet_manager.cc:1397] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20250110 20:51:37.713603 1611 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: No bootstrap required, opened a new log
I20250110 20:51:37.713764 1612 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25: No bootstrap required, opened a new log
I20250110 20:51:37.713856 1611 ts_tablet_manager.cc:1397] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250110 20:51:37.713990 1612 ts_tablet_manager.cc:1397] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250110 20:51:37.719754 1612 raft_consensus.cc:357] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.720165 1612 raft_consensus.cc:383] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:37.720278 1612 raft_consensus.cc:738] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e74bc04852a54c5db0d7d7f5d3fe5e25, State: Initialized, Role: FOLLOWER
I20250110 20:51:37.720834 1612 consensus_queue.cc:260] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.721751 1462 heartbeater.cc:502] Master 127.0.104.62:33135 was elected leader, sending a full tablet report...
I20250110 20:51:37.721673 1610 raft_consensus.cc:357] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.722074 1610 raft_consensus.cc:383] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:37.721879 1611 raft_consensus.cc:357] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.722172 1610 raft_consensus.cc:738] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9021cdbe740d4eaf834b08c85ecce67d, State: Initialized, Role: FOLLOWER
I20250110 20:51:37.722220 1611 raft_consensus.cc:383] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:37.722306 1611 raft_consensus.cc:738] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbb9c58f778d4c65a3660ac7479651ba, State: Initialized, Role: FOLLOWER
I20250110 20:51:37.722891 1610 consensus_queue.cc:260] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.722896 1611 consensus_queue.cc:260] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.725791 1612 ts_tablet_manager.cc:1428] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20250110 20:51:37.727190 1611 ts_tablet_manager.cc:1428] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: Time spent starting tablet: real 0.013s user 0.009s sys 0.006s
I20250110 20:51:37.727871 1331 heartbeater.cc:502] Master 127.0.104.62:33135 was elected leader, sending a full tablet report...
I20250110 20:51:37.727912 1592 heartbeater.cc:502] Master 127.0.104.62:33135 was elected leader, sending a full tablet report...
I20250110 20:51:37.728605 1610 ts_tablet_manager.cc:1428] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20250110 20:51:37.776836 1332 tablet.cc:2367] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:51:37.779928 1618 raft_consensus.cc:491] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:51:37.780144 1618 raft_consensus.cc:513] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.781672 1618 leader_election.cc:290] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e74bc04852a54c5db0d7d7f5d3fe5e25 (127.0.104.2:43219), bbb9c58f778d4c65a3660ac7479651ba (127.0.104.1:39637)
I20250110 20:51:37.788311 1286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b422b3f077f43a6bef8b5a158fc60b9" candidate_uuid: "9021cdbe740d4eaf834b08c85ecce67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb9c58f778d4c65a3660ac7479651ba" is_pre_election: true
I20250110 20:51:37.788318 1417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b422b3f077f43a6bef8b5a158fc60b9" candidate_uuid: "9021cdbe740d4eaf834b08c85ecce67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" is_pre_election: true
I20250110 20:51:37.788710 1286 raft_consensus.cc:2463] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9021cdbe740d4eaf834b08c85ecce67d in term 0.
I20250110 20:51:37.788762 1417 raft_consensus.cc:2463] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9021cdbe740d4eaf834b08c85ecce67d in term 0.
I20250110 20:51:37.789253 1480 leader_election.cc:304] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9021cdbe740d4eaf834b08c85ecce67d, e74bc04852a54c5db0d7d7f5d3fe5e25; no voters:
I20250110 20:51:37.789590 1618 raft_consensus.cc:2798] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:51:37.789700 1618 raft_consensus.cc:491] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:51:37.789785 1618 raft_consensus.cc:3054] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:37.791716 1618 raft_consensus.cc:513] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.792220 1618 leader_election.cc:290] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [CANDIDATE]: Term 1 election: Requested vote from peers e74bc04852a54c5db0d7d7f5d3fe5e25 (127.0.104.2:43219), bbb9c58f778d4c65a3660ac7479651ba (127.0.104.1:39637)
I20250110 20:51:37.792560 1417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b422b3f077f43a6bef8b5a158fc60b9" candidate_uuid: "9021cdbe740d4eaf834b08c85ecce67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25"
I20250110 20:51:37.792732 1417 raft_consensus.cc:3054] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:37.792742 1286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b422b3f077f43a6bef8b5a158fc60b9" candidate_uuid: "9021cdbe740d4eaf834b08c85ecce67d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bbb9c58f778d4c65a3660ac7479651ba"
I20250110 20:51:37.792883 1286 raft_consensus.cc:3054] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:37.794673 1417 raft_consensus.cc:2463] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9021cdbe740d4eaf834b08c85ecce67d in term 1.
I20250110 20:51:37.794731 1286 raft_consensus.cc:2463] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9021cdbe740d4eaf834b08c85ecce67d in term 1.
I20250110 20:51:37.795044 1480 leader_election.cc:304] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9021cdbe740d4eaf834b08c85ecce67d, e74bc04852a54c5db0d7d7f5d3fe5e25; no voters:
I20250110 20:51:37.795320 1618 raft_consensus.cc:2798] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:51:37.795703 1618 raft_consensus.cc:695] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 1 LEADER]: Becoming Leader. State: Replica: 9021cdbe740d4eaf834b08c85ecce67d, State: Running, Role: LEADER
I20250110 20:51:37.796098 1618 consensus_queue.cc:237] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:37.800222 1145 catalog_manager.cc:5526] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d reported cstate change: term changed from 0 to 1, leader changed from <none> to 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3). New cstate: current_term: 1 leader_uuid: "9021cdbe740d4eaf834b08c85ecce67d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } health_report { overall_health: UNKNOWN } } }
I20250110 20:51:37.835237 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:51:37.837551 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bbb9c58f778d4c65a3660ac7479651ba to finish bootstrapping
I20250110 20:51:37.845062 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e74bc04852a54c5db0d7d7f5d3fe5e25 to finish bootstrapping
I20250110 20:51:37.851325 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9021cdbe740d4eaf834b08c85ecce67d to finish bootstrapping
W20250110 20:51:37.867647 1593 tablet.cc:2367] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:51:37.881961 1286 raft_consensus.cc:1270] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 1 FOLLOWER]: Refusing update from remote peer 9021cdbe740d4eaf834b08c85ecce67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:51:37.881961 1417 raft_consensus.cc:1270] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 1 FOLLOWER]: Refusing update from remote peer 9021cdbe740d4eaf834b08c85ecce67d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:51:37.882851 1620 consensus_queue.cc:1035] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:51:37.883107 1618 consensus_queue.cc:1035] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:51:38.030790 1266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:51:38.030829 1527 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:51:38.031625 1397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6d626cafb8cf4ed5813a7c8afb66b34d | 127.0.104.62:33135 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
9021cdbe740d4eaf834b08c85ecce67d | 127.0.104.3:45405 | HEALTHY | <none> | 1 | 0
bbb9c58f778d4c65a3660ac7479651ba | 127.0.104.1:39637 | HEALTHY | <none> | 0 | 0
e74bc04852a54c5db0d7d7f5d3fe5e25 | 127.0.104.2:43219 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:39637
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:43219
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:45405
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:39637
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:43219
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:45405
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 20:51:38.083607 416 log_verifier.cc:126] Checking tablet 3b422b3f077f43a6bef8b5a158fc60b9
I20250110 20:51:38.116439 416 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3b422b3f077f43a6bef8b5a158fc60b9
I20250110 20:51:38.162560 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1465
I20250110 20:51:38.163349 1587 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:38.271338 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1465
I20250110 20:51:38.286585 1416 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3b422b3f077f43a6bef8b5a158fc60b9"
dest_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25"
from {username='slave'} at 127.0.0.1:56640
I20250110 20:51:38.286895 1416 raft_consensus.cc:491] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:51:38.287029 1416 raft_consensus.cc:3054] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:51:38.288901 1416 raft_consensus.cc:513] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:38.290273 1416 leader_election.cc:290] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [CANDIDATE]: Term 2 election: Requested vote from peers 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405), bbb9c58f778d4c65a3660ac7479651ba (127.0.104.1:39637)
W20250110 20:51:38.292574 1351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.3:45405: connect: Connection refused (error 111)
W20250110 20:51:38.294293 1351 leader_election.cc:336] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405): Network error: Client connection negotiation failed: client connection to 127.0.104.3:45405: connect: Connection refused (error 111)
I20250110 20:51:38.297456 1285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b422b3f077f43a6bef8b5a158fc60b9" candidate_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "bbb9c58f778d4c65a3660ac7479651ba"
I20250110 20:51:38.297624 1285 raft_consensus.cc:3054] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 1 FOLLOWER]: Advancing to term 2
W20250110 20:51:38.298090 1596 meta_cache.cc:302] tablet 3b422b3f077f43a6bef8b5a158fc60b9: replica 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405) has failed: Network error: Client connection negotiation failed: client connection to 127.0.104.3:45405: connect: Connection refused (error 111)
I20250110 20:51:38.299515 1285 raft_consensus.cc:2463] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e74bc04852a54c5db0d7d7f5d3fe5e25 in term 2.
I20250110 20:51:38.299911 1353 leader_election.cc:304] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbb9c58f778d4c65a3660ac7479651ba, e74bc04852a54c5db0d7d7f5d3fe5e25; no voters: 9021cdbe740d4eaf834b08c85ecce67d
I20250110 20:51:38.300267 1616 raft_consensus.cc:2798] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:51:38.300733 1616 raft_consensus.cc:695] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [term 2 LEADER]: Becoming Leader. State: Replica: e74bc04852a54c5db0d7d7f5d3fe5e25, State: Running, Role: LEADER
I20250110 20:51:38.301258 1616 consensus_queue.cc:237] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:38.304666 1145 catalog_manager.cc:5526] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 reported cstate change: term changed from 1 to 2, leader changed from 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3) to e74bc04852a54c5db0d7d7f5d3fe5e25 (127.0.104.2). New cstate: current_term: 2 leader_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } health_report { overall_health: UNKNOWN } } }
I20250110 20:51:38.310329 1285 raft_consensus.cc:1270] T 3b422b3f077f43a6bef8b5a158fc60b9 P bbb9c58f778d4c65a3660ac7479651ba [term 2 FOLLOWER]: Refusing update from remote peer e74bc04852a54c5db0d7d7f5d3fe5e25: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 20:51:38.310850 1667 consensus_queue.cc:1035] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 20:51:38.311072 1351 consensus_peers.cc:487] T 3b422b3f077f43a6bef8b5a158fc60b9 P e74bc04852a54c5db0d7d7f5d3fe5e25 -> Peer 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405): Couldn't send request to peer 9021cdbe740d4eaf834b08c85ecce67d. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:45405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:51:38.313952 1626 mvcc.cc:204] Tried to move back new op lower bound from 7112877253873827840 to 7112877253846753280. Current Snapshot: MvccSnapshot[applied={T|T < 7112877253873827840}]
I20250110 20:51:38.317961 1625 mvcc.cc:204] Tried to move back new op lower bound from 7112877253873827840 to 7112877253846753280. Current Snapshot: MvccSnapshot[applied={T|T < 7112877253873827840}]
I20250110 20:51:38.357270 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:45405
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:33135
--webserver_port=45083
--webserver_interface=127.0.104.3
--builtin_ntp_servers=127.0.104.20:39061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20250110 20:51:38.651425 1672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:38.651674 1672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:38.651798 1672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:38.658970 1672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:38.659140 1672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:51:38.667690 1672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:45405
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=45083
--tserver_master_addrs=127.0.104.62:33135
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:38.669093 1672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:38.670681 1672 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:38.681511 1678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:38.681643 1677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:38.681674 1680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:38.682587 1672 server_base.cc:1034] running on GCE node
I20250110 20:51:38.683171 1672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:38.684098 1672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:38.685317 1672 hybrid_clock.cc:648] HybridClock initialized: now 1736542298685255 us; error 43 us; skew 500 ppm
I20250110 20:51:38.685619 1672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:38.687768 1672 webserver.cc:458] Webserver started at http://127.0.104.3:45083/ using document root <none> and password file <none>
I20250110 20:51:38.688477 1672 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:38.688599 1672 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:38.693521 1672 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20250110 20:51:38.696169 1686 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:38.697456 1672 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250110 20:51:38.697605 1672 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9021cdbe740d4eaf834b08c85ecce67d"
format_stamp: "Formatted at 2025-01-10 20:51:37 on dist-test-slave-1h8z"
I20250110 20:51:38.698354 1672 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:38.721397 1672 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:38.722174 1672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:38.722410 1672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:38.723315 1672 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:38.725040 1693 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250110 20:51:38.729820 1672 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250110 20:51:38.729928 1672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20250110 20:51:38.730026 1672 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250110 20:51:38.732698 1672 ts_tablet_manager.cc:610] Registered 1 tablets
I20250110 20:51:38.732774 1672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20250110 20:51:38.733116 1693 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Bootstrap starting.
I20250110 20:51:38.753527 1693 log.cc:826] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:38.765444 1672 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45405
I20250110 20:51:38.765470 1800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45405 every 8 connection(s)
I20250110 20:51:38.766590 1672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:51:38.770795 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1672
I20250110 20:51:38.782491 1801 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33135
I20250110 20:51:38.782904 1801 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:38.783821 1801 heartbeater.cc:510] Master 127.0.104.62:33135 requested a full tablet report, sending...
I20250110 20:51:38.786403 1145 ts_manager.cc:194] Re-registered known tserver with Master: 9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405)
I20250110 20:51:38.787890 1145 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:35911
I20250110 20:51:38.794186 1693 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 20:51:38.794781 1693 tablet_bootstrap.cc:492] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Bootstrap complete.
I20250110 20:51:38.795960 1693 ts_tablet_manager.cc:1397] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Time spent bootstrapping tablet: real 0.063s user 0.048s sys 0.011s
I20250110 20:51:38.800217 1693 raft_consensus.cc:357] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:38.800915 1693 raft_consensus.cc:738] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9021cdbe740d4eaf834b08c85ecce67d, State: Initialized, Role: FOLLOWER
I20250110 20:51:38.801524 1693 consensus_queue.cc:260] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9021cdbe740d4eaf834b08c85ecce67d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45405 } } peers { permanent_uuid: "e74bc04852a54c5db0d7d7f5d3fe5e25" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43219 } } peers { permanent_uuid: "bbb9c58f778d4c65a3660ac7479651ba" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39637 } }
I20250110 20:51:38.802497 1801 heartbeater.cc:502] Master 127.0.104.62:33135 was elected leader, sending a full tablet report...
I20250110 20:51:38.803123 1693 ts_tablet_manager.cc:1428] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250110 20:51:38.873078 1266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:51:38.876565 1397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:51:38.879949 1735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6d626cafb8cf4ed5813a7c8afb66b34d | 127.0.104.62:33135 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
9021cdbe740d4eaf834b08c85ecce67d | 127.0.104.3:45405 | HEALTHY | <none> | 0 | 0
bbb9c58f778d4c65a3660ac7479651ba | 127.0.104.1:39637 | HEALTHY | <none> | 0 | 0
e74bc04852a54c5db0d7d7f5d3fe5e25 | 127.0.104.2:43219 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:39637
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:43219
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:45405
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:39637
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:43219
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:45405
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3b422b3f077f43a6bef8b5a158fc60b9 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405): RUNNING
e74bc04852a54c5db0d7d7f5d3fe5e25 (127.0.104.2:43219): RUNNING [LEADER]
bbb9c58f778d4c65a3660ac7479651ba (127.0.104.1:39637): RUNNING
All reported replicas are:
A = 9021cdbe740d4eaf834b08c85ecce67d
B = e74bc04852a54c5db0d7d7f5d3fe5e25
C = bbb9c58f778d4c65a3660ac7479651ba
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250110 20:51:38.926055 416 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250110 20:51:39.146550 1266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:51:39.147284 1397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:51:39.147433 1735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:51:39.160705 1755 raft_consensus.cc:3054] T 3b422b3f077f43a6bef8b5a158fc60b9 P 9021cdbe740d4eaf834b08c85ecce67d [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:51:39.200274 1856 mvcc.cc:204] Tried to move back new op lower bound from 7112877253873827840 to 7112877253846753280. Current Snapshot: MvccSnapshot[applied={T|T < 7112877253873827840}]
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6d626cafb8cf4ed5813a7c8afb66b34d | 127.0.104.62:33135 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
9021cdbe740d4eaf834b08c85ecce67d | 127.0.104.3:45405 | HEALTHY | <none> | 0 | 0
bbb9c58f778d4c65a3660ac7479651ba | 127.0.104.1:39637 | HEALTHY | <none> | 0 | 0
e74bc04852a54c5db0d7d7f5d3fe5e25 | 127.0.104.2:43219 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:39637
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:43219
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:45405
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:39637
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:43219
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:45405
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 3b422b3f077f43a6bef8b5a158fc60b9 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
9021cdbe740d4eaf834b08c85ecce67d (127.0.104.3:45405): RUNNING
e74bc04852a54c5db0d7d7f5d3fe5e25 (127.0.104.2:43219): RUNNING [LEADER]
bbb9c58f778d4c65a3660ac7479651ba (127.0.104.1:39637): RUNNING
All reported replicas are:
A = 9021cdbe740d4eaf834b08c85ecce67d
B = e74bc04852a54c5db0d7d7f5d3fe5e25
C = bbb9c58f778d4c65a3660ac7479651ba
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A B* C | | | Yes
A | A B C | 1 | -1 | Yes
B | A B* C | 2 | -1 | Yes
C | A B* C | 2 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250110 20:51:39.211273 416 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250110 20:51:39.505590 1735 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:51:39.509747 1266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:51:39.513480 1397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
6d626cafb8cf4ed5813a7c8afb66b34d | 127.0.104.62:33135 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
9021cdbe740d4eaf834b08c85ecce67d | 127.0.104.3:45405 | HEALTHY | <none> | 0 | 0
bbb9c58f778d4c65a3660ac7479651ba | 127.0.104.1:39637 | HEALTHY | <none> | 0 | 0
e74bc04852a54c5db0d7d7f5d3fe5e25 | 127.0.104.2:43219 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:39637
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:43219
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:45405
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:39637
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:43219
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:45405
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39061 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 20:51:39.553391 416 log_verifier.cc:126] Checking tablet 3b422b3f077f43a6bef8b5a158fc60b9
I20250110 20:51:39.589361 416 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 3b422b3f077f43a6bef8b5a158fc60b9
W20250110 20:51:40.146225 416 scanner-internal.cc:458] Time spent opening tablet: real 0.546s user 0.001s sys 0.000s
I20250110 20:51:40.147930 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1204
I20250110 20:51:40.149024 1325 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:40.249532 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1204
I20250110 20:51:40.271847 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1335
I20250110 20:51:40.272760 1457 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:40.368005 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1335
I20250110 20:51:40.381734 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1672
I20250110 20:51:40.389160 1795 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:40.503060 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1672
I20250110 20:51:40.518172 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1114
I20250110 20:51:40.519198 1175 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:51:40.627084 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1114
2025-01-10T20:51:40Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (4822 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20250110 20:51:40.664484 416 test_util.cc:274] Using random seed: 1123545729
I20250110 20:51:40.666025 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:51:40.666095 416 ts_itest-base.cc:113] --------------
I20250110 20:51:40.666138 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:51:40.666195 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:51:40.666230 416 ts_itest-base.cc:116] --------------
2025-01-10T20:51:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:51:40Z Disabled control of system clock
I20250110 20:51:40.687254 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35383
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35383 with env {}
W20250110 20:51:40.985839 1905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:40.986117 1905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:40.986238 1905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:40.993569 1905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:51:40.993666 1905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:40.993736 1905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:51:40.993813 1905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:51:41.002414 1905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35383
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:41.003858 1905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:41.005414 1905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:41.014875 1910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:41.014916 1913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:41.014940 1911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:41.015888 1905 server_base.cc:1034] running on GCE node
I20250110 20:51:41.016546 1905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:41.017768 1905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:41.018963 1905 hybrid_clock.cc:648] HybridClock initialized: now 1736542301018908 us; error 35 us; skew 500 ppm
I20250110 20:51:41.019248 1905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:41.025682 1905 webserver.cc:458] Webserver started at http://127.0.104.62:40045/ using document root <none> and password file <none>
I20250110 20:51:41.026315 1905 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:41.026444 1905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:41.026762 1905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:41.030282 1905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7f090285d86b4c2593b052bc593f8678"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.031030 1905 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7f090285d86b4c2593b052bc593f8678"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.035720 1905 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20250110 20:51:41.038587 1919 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:41.039798 1905 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250110 20:51:41.039937 1905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7f090285d86b4c2593b052bc593f8678"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.040082 1905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:41.056080 1905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:41.056778 1905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:41.057000 1905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:41.071555 1905 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35383
I20250110 20:51:41.071589 1970 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35383 every 8 connection(s)
I20250110 20:51:41.072732 1905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:51:41.076102 1971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:41.080109 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1905
I20250110 20:51:41.080292 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:51:41.085918 1971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678: Bootstrap starting.
I20250110 20:51:41.089572 1971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:41.090651 1971 log.cc:826] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:41.093268 1971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678: No bootstrap required, opened a new log
I20250110 20:51:41.099332 1971 raft_consensus.cc:357] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f090285d86b4c2593b052bc593f8678" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35383 } }
I20250110 20:51:41.099660 1971 raft_consensus.cc:383] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:41.099735 1971 raft_consensus.cc:738] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f090285d86b4c2593b052bc593f8678, State: Initialized, Role: FOLLOWER
I20250110 20:51:41.100206 1971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f090285d86b4c2593b052bc593f8678" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35383 } }
I20250110 20:51:41.100437 1971 raft_consensus.cc:397] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:51:41.100549 1971 raft_consensus.cc:491] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:51:41.100703 1971 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:41.102541 1971 raft_consensus.cc:513] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f090285d86b4c2593b052bc593f8678" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35383 } }
I20250110 20:51:41.102975 1971 leader_election.cc:304] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f090285d86b4c2593b052bc593f8678; no voters:
I20250110 20:51:41.103499 1971 leader_election.cc:290] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:51:41.103610 1976 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:51:41.104564 1976 raft_consensus.cc:695] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [term 1 LEADER]: Becoming Leader. State: Replica: 7f090285d86b4c2593b052bc593f8678, State: Running, Role: LEADER
I20250110 20:51:41.105078 1976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f090285d86b4c2593b052bc593f8678" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35383 } }
I20250110 20:51:41.105284 1971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:51:41.107707 1977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f090285d86b4c2593b052bc593f8678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f090285d86b4c2593b052bc593f8678" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35383 } } }
I20250110 20:51:41.107762 1978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f090285d86b4c2593b052bc593f8678. Latest consensus state: current_term: 1 leader_uuid: "7f090285d86b4c2593b052bc593f8678" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f090285d86b4c2593b052bc593f8678" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35383 } } }
I20250110 20:51:41.108021 1977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [sys.catalog]: This master's current role is: LEADER
I20250110 20:51:41.108052 1978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678 [sys.catalog]: This master's current role is: LEADER
I20250110 20:51:41.109598 1982 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:51:41.114426 1982 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:51:41.123154 1982 catalog_manager.cc:1348] Generated new cluster ID: 3bbe6bee8b714eb69df7d3ae54978004
I20250110 20:51:41.123247 1982 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:51:41.135138 1982 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:51:41.136140 1982 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:51:41.146191 1982 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 7f090285d86b4c2593b052bc593f8678: Generated new TSK 0
I20250110 20:51:41.146975 1982 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:51:41.161247 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35383
--builtin_ntp_servers=127.0.104.20:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20250110 20:51:41.466454 1995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:41.466708 1995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:41.466859 1995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:41.474081 1995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:41.474256 1995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:51:41.482939 1995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:35383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:41.484326 1995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:41.485859 1995 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:41.496515 2000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:41.496558 2003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:41.496588 2001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:41.497306 1995 server_base.cc:1034] running on GCE node
I20250110 20:51:41.497840 1995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:41.498647 1995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:41.499850 1995 hybrid_clock.cc:648] HybridClock initialized: now 1736542301499797 us; error 34 us; skew 500 ppm
I20250110 20:51:41.500113 1995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:41.502022 1995 webserver.cc:458] Webserver started at http://127.0.104.1:42233/ using document root <none> and password file <none>
I20250110 20:51:41.502656 1995 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:41.502789 1995 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:41.503065 1995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:41.505383 1995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.506055 1995 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.510718 1995 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250110 20:51:41.513587 2009 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:41.514712 1995 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250110 20:51:41.514907 1995 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.515059 1995 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:41.538071 1995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:41.538815 1995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:41.539053 1995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:41.539903 1995 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:41.541260 1995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:41.541326 1995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:41.541433 1995 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:41.541486 1995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:41.569329 1995 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35577
I20250110 20:51:41.569375 2121 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35577 every 8 connection(s)
I20250110 20:51:41.570498 1995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:51:41.578600 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 1995
I20250110 20:51:41.578790 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:51:41.582010 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35383
--builtin_ntp_servers=127.0.104.20:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250110 20:51:41.582134 2122 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35383
I20250110 20:51:41.582378 2122 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:41.583122 2122 heartbeater.cc:510] Master 127.0.104.62:35383 requested a full tablet report, sending...
I20250110 20:51:41.585157 1936 ts_manager.cc:194] Registered new tserver with Master: 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577)
I20250110 20:51:41.586922 1936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:47245
W20250110 20:51:41.883353 2125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:41.883666 2125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:41.883781 2125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:41.891633 2125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:41.891875 2125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:51:41.901216 2125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:35383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:41.902803 2125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:41.904497 2125 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:41.915856 2130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:41.915974 2133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:41.915853 2131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:41.916708 2125 server_base.cc:1034] running on GCE node
I20250110 20:51:41.917320 2125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:41.918313 2125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:41.919564 2125 hybrid_clock.cc:648] HybridClock initialized: now 1736542301919481 us; error 58 us; skew 500 ppm
I20250110 20:51:41.919900 2125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:41.922286 2125 webserver.cc:458] Webserver started at http://127.0.104.2:35687/ using document root <none> and password file <none>
I20250110 20:51:41.923408 2125 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:41.923578 2125 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:41.924062 2125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:41.927203 2125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "555db90ad09d47708f85e4658c394115"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.928155 2125 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "555db90ad09d47708f85e4658c394115"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.934109 2125 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250110 20:51:41.937781 2139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:41.939421 2125 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:51:41.939625 2125 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "555db90ad09d47708f85e4658c394115"
format_stamp: "Formatted at 2025-01-10 20:51:41 on dist-test-slave-1h8z"
I20250110 20:51:41.939827 2125 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:41.967216 2125 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:41.968170 2125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:41.968456 2125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:41.969493 2125 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:41.971156 2125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:41.971237 2125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:41.971365 2125 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:41.971418 2125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:42.001158 2125 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:40905
I20250110 20:51:42.001219 2251 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:40905 every 8 connection(s)
I20250110 20:51:42.002533 2125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:51:42.008740 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 2125
I20250110 20:51:42.008924 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:51:42.012099 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35383
--builtin_ntp_servers=127.0.104.20:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250110 20:51:42.015966 2252 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35383
I20250110 20:51:42.016273 2252 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:42.017109 2252 heartbeater.cc:510] Master 127.0.104.62:35383 requested a full tablet report, sending...
I20250110 20:51:42.018530 1936 ts_manager.cc:194] Registered new tserver with Master: 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905)
I20250110 20:51:42.019553 1936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:45053
W20250110 20:51:42.341759 2255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:51:42.342057 2255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:51:42.342152 2255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:51:42.350013 2255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:51:42.350221 2255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:51:42.359426 2255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:35383
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:51:42.361019 2255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:51:42.362679 2255 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:51:42.374269 2260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:42.374325 2263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:51:42.374305 2261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:51:42.375134 2255 server_base.cc:1034] running on GCE node
I20250110 20:51:42.375736 2255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:51:42.376696 2255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:51:42.377923 2255 hybrid_clock.cc:648] HybridClock initialized: now 1736542302377842 us; error 52 us; skew 500 ppm
I20250110 20:51:42.378211 2255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:51:42.380503 2255 webserver.cc:458] Webserver started at http://127.0.104.3:41413/ using document root <none> and password file <none>
I20250110 20:51:42.381186 2255 fs_manager.cc:362] Metadata directory not provided
I20250110 20:51:42.381284 2255 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:51:42.381561 2255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:51:42.384234 2255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "95d4d0dd0b954191b938a467d41914e7"
format_stamp: "Formatted at 2025-01-10 20:51:42 on dist-test-slave-1h8z"
I20250110 20:51:42.384979 2255 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "95d4d0dd0b954191b938a467d41914e7"
format_stamp: "Formatted at 2025-01-10 20:51:42 on dist-test-slave-1h8z"
I20250110 20:51:42.390338 2255 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 20:51:42.394444 2269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:42.395980 2255 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250110 20:51:42.396138 2255 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "95d4d0dd0b954191b938a467d41914e7"
format_stamp: "Formatted at 2025-01-10 20:51:42 on dist-test-slave-1h8z"
I20250110 20:51:42.396308 2255 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:51:42.416330 2255 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:51:42.417124 2255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:51:42.417354 2255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:51:42.418208 2255 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:51:42.419734 2255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:51:42.419803 2255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:42.419891 2255 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:51:42.419931 2255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:51:42.449592 2255 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44233
I20250110 20:51:42.449626 2381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44233 every 8 connection(s)
I20250110 20:51:42.450950 2255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:51:42.451875 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 2255
I20250110 20:51:42.452044 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:51:42.464226 2382 heartbeater.cc:346] Connected to a master server at 127.0.104.62:35383
I20250110 20:51:42.464540 2382 heartbeater.cc:463] Registering TS with master...
I20250110 20:51:42.465389 2382 heartbeater.cc:510] Master 127.0.104.62:35383 requested a full tablet report, sending...
I20250110 20:51:42.466917 1935 ts_manager.cc:194] Registered new tserver with Master: 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:42.467818 1935 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:48439
I20250110 20:51:42.467959 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:51:42.471458 416 test_util.cc:274] Using random seed: 1125352715
I20250110 20:51:42.489151 1935 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48990:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:51:42.490689 1935 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:51:42.529026 2187 tablet_service.cc:1467] Processing CreateTablet for tablet 66baaff8be8d49be99ab53b640904c2c (DEFAULT_TABLE table=test-workload [id=81ad58d7b1a44f989cb245bc4020edeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:42.529732 2317 tablet_service.cc:1467] Processing CreateTablet for tablet 66baaff8be8d49be99ab53b640904c2c (DEFAULT_TABLE table=test-workload [id=81ad58d7b1a44f989cb245bc4020edeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:42.529732 2057 tablet_service.cc:1467] Processing CreateTablet for tablet 66baaff8be8d49be99ab53b640904c2c (DEFAULT_TABLE table=test-workload [id=81ad58d7b1a44f989cb245bc4020edeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:51:42.530429 2187 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66baaff8be8d49be99ab53b640904c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:42.531069 2057 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66baaff8be8d49be99ab53b640904c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:42.531069 2317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66baaff8be8d49be99ab53b640904c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:51:42.541549 2400 tablet_bootstrap.cc:492] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115: Bootstrap starting.
I20250110 20:51:42.544049 2401 tablet_bootstrap.cc:492] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1: Bootstrap starting.
I20250110 20:51:42.545408 2400 tablet_bootstrap.cc:654] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:42.545679 2402 tablet_bootstrap.cc:492] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7: Bootstrap starting.
I20250110 20:51:42.546924 2400 log.cc:826] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:42.547901 2401 tablet_bootstrap.cc:654] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:42.548974 2402 tablet_bootstrap.cc:654] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7: Neither blocks nor log segments found. Creating new log.
I20250110 20:51:42.549083 2401 log.cc:826] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:42.550009 2402 log.cc:826] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7: Log is configured to *not* fsync() on all Append() calls
I20250110 20:51:42.550532 2400 tablet_bootstrap.cc:492] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115: No bootstrap required, opened a new log
I20250110 20:51:42.550879 2400 ts_tablet_manager.cc:1397] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250110 20:51:42.551787 2401 tablet_bootstrap.cc:492] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1: No bootstrap required, opened a new log
I20250110 20:51:42.552038 2401 ts_tablet_manager.cc:1397] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20250110 20:51:42.552130 2402 tablet_bootstrap.cc:492] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7: No bootstrap required, opened a new log
I20250110 20:51:42.552343 2402 ts_tablet_manager.cc:1397] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20250110 20:51:42.558530 2402 raft_consensus.cc:357] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.558941 2402 raft_consensus.cc:383] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:42.559065 2402 raft_consensus.cc:738] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95d4d0dd0b954191b938a467d41914e7, State: Initialized, Role: FOLLOWER
I20250110 20:51:42.559695 2402 consensus_queue.cc:260] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.559597 2400 raft_consensus.cc:357] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.560024 2400 raft_consensus.cc:383] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:42.560176 2400 raft_consensus.cc:738] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555db90ad09d47708f85e4658c394115, State: Initialized, Role: FOLLOWER
I20250110 20:51:42.560369 2401 raft_consensus.cc:357] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.560940 2401 raft_consensus.cc:383] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:51:42.561064 2401 raft_consensus.cc:738] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea09a0a2f2549a4a06723c0478f03f1, State: Initialized, Role: FOLLOWER
I20250110 20:51:42.560870 2400 consensus_queue.cc:260] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.561376 2402 ts_tablet_manager.cc:1428] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20250110 20:51:42.566370 2382 heartbeater.cc:502] Master 127.0.104.62:35383 was elected leader, sending a full tablet report...
I20250110 20:51:42.566943 2400 ts_tablet_manager.cc:1428] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115: Time spent starting tablet: real 0.016s user 0.011s sys 0.003s
I20250110 20:51:42.567559 2252 heartbeater.cc:502] Master 127.0.104.62:35383 was elected leader, sending a full tablet report...
I20250110 20:51:42.568390 2401 consensus_queue.cc:260] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.570907 2408 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:51:42.571216 2408 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.572817 2406 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:51:42.573071 2406 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.577390 2401 ts_tablet_manager.cc:1428] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1: Time spent starting tablet: real 0.025s user 0.011s sys 0.003s
I20250110 20:51:42.578094 2406 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905)
W20250110 20:51:42.584100 2270 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.1:35577 timed out after 0.008s (SENT)
I20250110 20:51:42.587136 2409 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:51:42.587446 2409 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.589614 2409 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:42.577708 2408 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
W20250110 20:51:42.598059 2011 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.2:40905 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20250110 20:51:42.598901 2122 heartbeater.cc:502] Master 127.0.104.62:35383 was elected leader, sending a full tablet report...
W20250110 20:51:42.602939 2140 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.1:35577 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20250110 20:51:42.610368 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:42.611060 2207 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 0.
I20250110 20:51:42.611830 2271 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7; no voters:
I20250110 20:51:42.612402 2407 raft_consensus.cc:2798] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:51:42.612543 2407 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:51:42.612684 2407 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:42.615299 2076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:42.615798 2076 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 0.
I20250110 20:51:42.616393 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:42.615332 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:42.616957 2077 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 0.
I20250110 20:51:42.618124 2407 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
W20250110 20:51:42.618381 2270 leader_election.cc:336] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Timed out: RequestConsensusVote RPC to 127.0.104.1:35577 timed out after 0.008s (SENT)
I20250110 20:51:42.618697 2337 raft_consensus.cc:2388] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ea09a0a2f2549a4a06723c0478f03f1 in current term 1: Already voted for candidate 95d4d0dd0b954191b938a467d41914e7 in this term.
I20250110 20:51:42.620899 2407 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 1 election: Requested vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905)
I20250110 20:51:42.621243 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115"
I20250110 20:51:42.621475 2207 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:51:42.622028 2011 leader_election.cc:336] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905): Timed out: RequestConsensusVote RPC to 127.0.104.2:40905 timed out after 0.006s (SENT)
I20250110 20:51:42.622200 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:42.622571 2408 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:51:42.624752 2207 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 1.
I20250110 20:51:42.625171 2271 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7; no voters:
I20250110 20:51:42.625222 2408 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:51:42.625763 2407 raft_consensus.cc:2798] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:51:42.626583 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
I20250110 20:51:42.626785 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:42.627069 2207 raft_consensus.cc:2388] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ea09a0a2f2549a4a06723c0478f03f1 in current term 1: Already voted for candidate 95d4d0dd0b954191b938a467d41914e7 in this term.
I20250110 20:51:42.627607 2407 raft_consensus.cc:695] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 LEADER]: Becoming Leader. State: Replica: 95d4d0dd0b954191b938a467d41914e7, State: Running, Role: LEADER
I20250110 20:51:42.628196 2407 consensus_queue.cc:237] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:42.629756 2076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:42.629778 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
I20250110 20:51:42.632428 2077 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 1.
I20250110 20:51:42.633987 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
W20250110 20:51:42.634582 2143 leader_election.cc:336] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233): Timed out: RequestConsensusVote RPC to 127.0.104.3:44233 timed out after 0.007s (SENT)
I20250110 20:51:42.634827 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:42.634601 1935 catalog_manager.cc:5526] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "95d4d0dd0b954191b938a467d41914e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } } }
I20250110 20:51:42.635310 2410 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:51:42.635563 2077 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.3:36091 (request call id 2) took 8 ms (client timeout 6 ms). Trace:
W20250110 20:51:42.635701 2077 rpcz_store.cc:269] 0110 20:51:42.626422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:42.626471 (+ 49us) service_pool.cc:225] Handling call
0110 20:51:42.635144 (+ 8673us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250110 20:51:42.664743 2425 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:51:42.664837 2425 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250110 20:51:42.809268 2036 tablet_service.cc:2970] error setting up scanner d8ae297e4f0e4d9386118697ff6187ad with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.145s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:42.809237 2037 tablet_service.cc:2970] error setting up scanner c6e6d364ae7048a8b0568960080e2f1a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.145s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
W20250110 20:51:42.827560 2036 tablet_service.cc:2970] error setting up scanner a8b9eeb6a0de4830a51af1f951902663 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.163s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
W20250110 20:51:42.829274 2036 tablet_service.cc:2970] error setting up scanner 35d2732af02e4b628e413e8d1c69a1ee with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.165s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:42.853875 2036 tablet_service.cc:2970] error setting up scanner c1cef1d420664d35be10dffac1e1faad with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.189s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:42.862151 2036 tablet_service.cc:2970] error setting up scanner c085c626d5f2499fa859d7eb78e50509 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.198s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
W20250110 20:51:42.896970 2036 tablet_service.cc:2970] error setting up scanner 880ec2dafeab46729290cfbd3d53dd4d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.233s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:42.904808 2036 tablet_service.cc:2970] error setting up scanner 918c3d378e1b4e83bc5f2709334d7918 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.240s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
W20250110 20:51:43.019940 2036 tablet_service.cc:2970] error setting up scanner 2d44b50e5f1b49a48f6d18b8e8e2dc95 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.356s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:43.059473 2036 tablet_service.cc:2970] error setting up scanner 4a9112caaa4445f589696c14aad9eb5d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.395s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
W20250110 20:51:43.198489 2036 tablet_service.cc:2970] error setting up scanner ee94ca0b830441afaa2aff3ef3da10af with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.534s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:43.302178 2036 tablet_service.cc:2970] error setting up scanner f1b77a4a22c04343b108dba57bf62e11 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.638s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:43.455808 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:43.456034 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:43.456818 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:43.457258 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:43.457248 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:43.457947 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:43.458248 2434 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:51:43.537974 2207 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:51:43.538071 2207 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250110 20:51:43.538178 2075 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:51:43.538292 2075 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250110 20:51:43.617408 2036 tablet_service.cc:2970] error setting up scanner 878b0a39aead4014ac7b6a67a445e7a0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.953s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:43.849115 2036 tablet_service.cc:2970] error setting up scanner 2d2155d8ab36497fb00dcec73cdca15c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.185s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:44.019414 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:44.019670 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:44.020689 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:44.021270 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:44.021488 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:44.022045 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:44.022351 2434 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:51:44.024724 2438 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=1, attempt_no=0}) took 1365 ms. Trace:
I20250110 20:51:44.024868 2438 rpcz_store.cc:276] 0110 20:51:42.658905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:42.658973 (+ 68us) service_pool.cc:225] Handling call
0110 20:51:44.024657 (+1365684us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:42.662472 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:42.662711 (+ 239us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:42.662733 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:51:42.662759 (+ 26us) tablet.cc:659] Decoding operations
0110 20:51:42.663366 (+ 607us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:42.663454 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:42.663467 (+ 13us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:42.664495 (+ 1028us) tablet.cc:698] Row locks acquired
0110 20:51:42.664499 (+ 4us) write_op.cc:260] PREPARE: finished
0110 20:51:42.664639 (+ 140us) write_op.cc:270] Start()
0110 20:51:43.535162 (+870523us) write_op.cc:276] Timestamp: P: 1736542302664571 usec, L: 0
0110 20:51:43.535170 (+ 8us) op_driver.cc:348] REPLICATION: starting
0110 20:51:43.535569 (+ 399us) log.cc:844] Serialized 345 byte log entry
0110 20:51:44.021026 (+485457us) op_driver.cc:464] REPLICATION: finished
0110 20:51:44.022620 (+ 1594us) write_op.cc:301] APPLY: starting
0110 20:51:44.022698 (+ 78us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:44.023017 (+ 319us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:44.023064 (+ 47us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:44.023512 (+ 448us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:44.023685 (+ 173us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:44.023707 (+ 22us) write_op.cc:312] APPLY: finished
0110 20:51:44.023996 (+ 289us) log.cc:844] Serialized 22 byte log entry
0110 20:51:44.024187 (+ 191us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:44.024256 (+ 69us) write_op.cc:454] Released schema lock
0110 20:51:44.024269 (+ 13us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":170752,"child_traces":[["op",{"apply.queue_time_us":1498,"num_ops":1,"prepare.queue_time_us":27,"prepare.run_cpu_time_us":3200,"prepare.run_wall_time_us":873256,"raft.queue_time_us":42,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":10,"replication_time_us":485781,"thread_start_us":1395,"threads_started":1}]]}
I20250110 20:51:44.261566 2435 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:44.261770 2435 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:44.263094 2435 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:44.263576 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:44.263844 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:44.264386 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:44.264649 2435 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:51:44.303674 2435 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:44.303870 2435 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:44.304713 2435 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:44.305106 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:44.305299 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:44.305622 2438 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=0, attempt_no=0}) took 1646 ms. Trace:
I20250110 20:51:44.305963 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:44.305734 2438 rpcz_store.cc:276] 0110 20:51:42.658726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:42.658813 (+ 87us) service_pool.cc:225] Handling call
0110 20:51:44.305561 (+1646748us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:43.535720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:43.535849 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:43.535856 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:51:43.535858 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:43.535954 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:43.535991 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:43.535994 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:43.536039 (+ 45us) tablet.cc:698] Row locks acquired
0110 20:51:43.536041 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:43.536085 (+ 44us) write_op.cc:270] Start()
0110 20:51:44.240217 (+704132us) write_op.cc:276] Timestamp: P: 1736542303536046 usec, L: 0
0110 20:51:44.240221 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:51:44.240404 (+ 183us) log.cc:844] Serialized 345 byte log entry
0110 20:51:44.304912 (+ 64508us) op_driver.cc:464] REPLICATION: finished
0110 20:51:44.305054 (+ 142us) write_op.cc:301] APPLY: starting
0110 20:51:44.305070 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:44.305113 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:44.305145 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:44.305227 (+ 82us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:44.305262 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:44.305269 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:44.305355 (+ 86us) log.cc:844] Serialized 22 byte log entry
0110 20:51:44.305437 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:44.305455 (+ 18us) write_op.cc:454] Released schema lock
0110 20:51:44.305462 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1473408,"child_traces":[["op",{"apply.queue_time_us":79,"mutex_wait_us":2620,"num_ops":1,"prepare.queue_time_us":873361,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":704753,"replication_time_us":64656}]]}
I20250110 20:51:44.306351 2435 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:51:44.555860 2036 tablet_service.cc:2970] error setting up scanner 093063aa090a4b138abfc5a8410abf7d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.533s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:44.767261 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:44.767519 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:44.768299 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:44.768810 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:44.768815 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:44.769390 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:45.180315 2435 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:45.180526 2435 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:45.181294 2435 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:45.181782 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:45.181813 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:45.182574 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:45.182974 2447 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=2, attempt_no=0}) took 1155 ms. Trace:
I20250110 20:51:45.183101 2447 rpcz_store.cc:276] 0110 20:51:44.027693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:44.027748 (+ 55us) service_pool.cc:225] Handling call
0110 20:51:45.182923 (+1155175us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:44.240499 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:44.240640 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:44.240647 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:51:44.240650 (+ 3us) tablet.cc:659] Decoding operations
0110 20:51:44.240727 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:44.240737 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:44.240740 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:44.240783 (+ 43us) tablet.cc:698] Row locks acquired
0110 20:51:44.240785 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:44.240835 (+ 50us) write_op.cc:270] Start()
0110 20:51:44.580983 (+340148us) write_op.cc:276] Timestamp: P: 1736542304240781 usec, L: 0
0110 20:51:44.580988 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:51:44.581212 (+ 224us) log.cc:844] Serialized 354 byte log entry
0110 20:51:45.181857 (+600645us) op_driver.cc:464] REPLICATION: finished
0110 20:51:45.182326 (+ 469us) write_op.cc:301] APPLY: starting
0110 20:51:45.182347 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:45.182392 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:45.182419 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:45.182508 (+ 89us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:45.182543 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:45.182551 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:51:45.182651 (+ 100us) log.cc:844] Serialized 22 byte log entry
0110 20:51:45.182793 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:45.182814 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:45.182821 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":384,"num_ops":1,"prepare.queue_time_us":212274,"prepare.run_cpu_time_us":716,"prepare.run_wall_time_us":340783,"replication_time_us":600833,"thread_start_us":290,"threads_started":1}]]}
I20250110 20:51:45.369740 2434 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:51:45.813459 2435 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:51:45.815681 2453 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=3, attempt_no=0}) took 1508 ms. Trace:
I20250110 20:51:45.815798 2453 rpcz_store.cc:276] 0110 20:51:44.307644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:44.307691 (+ 47us) service_pool.cc:225] Handling call
0110 20:51:45.815623 (+1507932us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:44.581350 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:44.581472 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:44.581479 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:51:44.581482 (+ 3us) tablet.cc:659] Decoding operations
0110 20:51:44.581576 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:44.581588 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:44.581593 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:44.581644 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:51:44.581646 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:44.581695 (+ 49us) write_op.cc:270] Start()
0110 20:51:44.997843 (+416148us) write_op.cc:276] Timestamp: P: 1736542304581653 usec, L: 0
0110 20:51:44.997848 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:51:44.998081 (+ 233us) log.cc:844] Serialized 354 byte log entry
0110 20:51:45.814630 (+816549us) op_driver.cc:464] REPLICATION: finished
0110 20:51:45.815179 (+ 549us) write_op.cc:301] APPLY: starting
0110 20:51:45.815198 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:45.815245 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:45.815283 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:45.815359 (+ 76us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:45.815386 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:45.815391 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:51:45.815464 (+ 73us) log.cc:844] Serialized 22 byte log entry
0110 20:51:45.815523 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:45.815541 (+ 18us) write_op.cc:454] Released schema lock
0110 20:51:45.815546 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":479,"num_ops":1,"prepare.queue_time_us":273140,"prepare.run_cpu_time_us":743,"prepare.run_wall_time_us":416802,"replication_time_us":816743,"thread_start_us":368,"threads_started":1}]]}
W20250110 20:51:45.836684 2036 tablet_service.cc:2970] error setting up scanner c39577b615c944feb5f9494f4aa01f5a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736542304836309 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736542304240781 usec, L: 0 Physical time difference: 0.596s: new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
I20250110 20:51:45.837160 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:46820 (request call id 15) took 1001 ms. Trace:
I20250110 20:51:45.837289 2036 rpcz_store.cc:276] 0110 20:51:44.835231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:44.835327 (+ 96us) service_pool.cc:225] Handling call
0110 20:51:44.835603 (+ 276us) tablet_service.cc:2780] Created scanner c39577b615c944feb5f9494f4aa01f5a for tablet 66baaff8be8d49be99ab53b640904c2c, query id is b8a8b18cc6c547ee99ba0169ad498ce5
0110 20:51:44.836326 (+ 723us) tablet_service.cc:2920] Creating iterator
0110 20:51:44.836358 (+ 32us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:51:45.836622 (+1000264us) tablet_service.cc:2945] Iterator created
0110 20:51:45.836637 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736542304836309 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736542304240781 usec, L: 0 Physical time difference: 0.596s
0110 20:51:45.837132 (+ 495us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250110 20:51:45.868798 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:45.869028 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:45.869783 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:45.870257 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:45.870894 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:45.871462 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
W20250110 20:51:46.094647 2036 tablet_service.cc:2970] error setting up scanner d9f1c065e579497a8762f94efe4ee8d8 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.224s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:46.412077 2435 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:46.412292 2435 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.412983 2434 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:51:46.413080 2435 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:46.413992 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:46.414062 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:46.414322 2075 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 555db90ad09d47708f85e4658c394115 in term 1.
I20250110 20:51:46.414898 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 555db90ad09d47708f85e4658c394115, 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:46.415841 2460 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=4, attempt_no=0}) took 1230 ms. Trace:
I20250110 20:51:46.416000 2460 rpcz_store.cc:276] 0110 20:51:45.184936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:45.185000 (+ 64us) service_pool.cc:225] Handling call
0110 20:51:46.415786 (+1230786us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:45.185482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:45.185581 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:45.185586 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:45.185588 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:45.185661 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:45.185670 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:45.185673 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:45.185712 (+ 39us) tablet.cc:698] Row locks acquired
0110 20:51:45.185714 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:45.185746 (+ 32us) write_op.cc:270] Start()
0110 20:51:45.564939 (+379193us) write_op.cc:276] Timestamp: P: 1736542305185704 usec, L: 0
0110 20:51:45.564944 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:51:45.565205 (+ 261us) log.cc:844] Serialized 354 byte log entry
0110 20:51:46.414561 (+849356us) op_driver.cc:464] REPLICATION: finished
0110 20:51:46.415217 (+ 656us) write_op.cc:301] APPLY: starting
0110 20:51:46.415234 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:46.415277 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:46.415336 (+ 59us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:46.415438 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:46.415473 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:46.415491 (+ 18us) write_op.cc:312] APPLY: finished
0110 20:51:46.415585 (+ 94us) log.cc:844] Serialized 22 byte log entry
0110 20:51:46.415664 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:46.415683 (+ 19us) write_op.cc:454] Released schema lock
0110 20:51:46.415690 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":584,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":379793,"replication_time_us":849584,"thread_start_us":1317,"threads_started":1}]]}
I20250110 20:51:46.762221 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:46.762428 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.764190 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:46.764565 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:46.765805 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:46.766280 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:46.766528 2434 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:51:46.873731 2435 raft_consensus.cc:2798] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:51:46.874053 2435 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:46.874202 2435 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:51:46.876804 2435 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.877218 2207 raft_consensus.cc:1235] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 2 FOLLOWER]: Rejecting Update request from peer 95d4d0dd0b954191b938a467d41914e7 for earlier term 1. Current term is 2. Ops: []
I20250110 20:51:46.877547 2435 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 election: Requested vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:46.878129 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7"
I20250110 20:51:46.878407 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
I20250110 20:51:46.878477 2458 consensus_queue.cc:1046] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 }, Status: INVALID_TERM, Last received: 1.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 20:51:46.878870 2459 raft_consensus.cc:3049] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:51:46.878962 2459 raft_consensus.cc:738] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 95d4d0dd0b954191b938a467d41914e7, State: Running, Role: LEADER
I20250110 20:51:46.878966 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:46.879200 2435 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250110 20:51:46.879529 2459 consensus_queue.cc:260] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.880018 2459 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:51:46.883368 2459 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:51:46.883518 2459 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.884111 2459 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905)
I20250110 20:51:46.884559 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:46.884608 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:46.884737 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.885035 2075 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 1.
I20250110 20:51:46.885344 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:46.885553 2270 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1; no voters:
I20250110 20:51:46.885710 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:46.885754 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:46.885824 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:46.885914 2207 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 2.
I20250110 20:51:46.886029 2337 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea09a0a2f2549a4a06723c0478f03f1 in term 2.
I20250110 20:51:46.886371 2459 raft_consensus.cc:2798] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:51:46.886487 2459 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:51:46.886479 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1; no voters:
I20250110 20:51:46.886582 2459 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:51:46.886765 2434 raft_consensus.cc:2798] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:51:46.886869 2434 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:46.886960 2434 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:51:46.888481 2459 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.888803 2434 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.889017 2459 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 3 election: Requested vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905)
I20250110 20:51:46.889534 2434 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 election: Requested vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:46.889540 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1"
I20250110 20:51:46.889794 2075 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:51:46.889773 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115"
I20250110 20:51:46.890030 2207 raft_consensus.cc:2388] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ea09a0a2f2549a4a06723c0478f03f1 in current term 2: Already voted for candidate 555db90ad09d47708f85e4658c394115 in this term.
I20250110 20:51:46.890327 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "95d4d0dd0b954191b938a467d41914e7" candidate_term: 3 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115"
I20250110 20:51:46.890496 2206 raft_consensus.cc:3054] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:51:46.892468 2075 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 3.
I20250110 20:51:46.892848 2270 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1; no voters:
I20250110 20:51:46.893083 2206 raft_consensus.cc:2463] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95d4d0dd0b954191b938a467d41914e7 in term 3.
I20250110 20:51:46.893157 2459 raft_consensus.cc:2798] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 3 FOLLOWER]: Leader election won for term 3
I20250110 20:51:46.893365 2459 raft_consensus.cc:695] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [term 3 LEADER]: Becoming Leader. State: Replica: 95d4d0dd0b954191b938a467d41914e7, State: Running, Role: LEADER
I20250110 20:51:46.893731 2459 consensus_queue.cc:237] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:46.893827 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7"
I20250110 20:51:46.894301 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:46.894590 2434 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250110 20:51:46.896289 2206 raft_consensus.cc:1270] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Refusing update from remote peer 95d4d0dd0b954191b938a467d41914e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20250110 20:51:46.896203 1936 catalog_manager.cc:5526] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "95d4d0dd0b954191b938a467d41914e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } health_report { overall_health: HEALTHY } } }
I20250110 20:51:46.896863 2075 raft_consensus.cc:1270] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Refusing update from remote peer 95d4d0dd0b954191b938a467d41914e7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 3 index: 8. (index mismatch)
I20250110 20:51:46.896903 2457 consensus_queue.cc:1035] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 20:51:46.897650 2457 consensus_queue.cc:1035] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20250110 20:51:47.210906 2425 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56756: Aborted: Op submitted in term 1 cannot be replicated in term 3
W20250110 20:51:47.216923 2036 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46820: Illegal state: replica 9ea09a0a2f2549a4a06723c0478f03f1 is not leader of this config: current role FOLLOWER
W20250110 20:51:47.242934 2167 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56042: Illegal state: replica 555db90ad09d47708f85e4658c394115 is not leader of this config: current role FOLLOWER
I20250110 20:51:47.456745 2470 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:47.457190 2470 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:47.458319 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:47.458328 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 8 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:47.459023 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:47.459698 2470 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:47.459939 2470 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:51:47.631909 2036 tablet_service.cc:2970] error setting up scanner a47ef17f974041cdb9741246bb154a38 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.419s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:47.882347 2425 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56756: Aborted: Op submitted in term 1 cannot be replicated in term 3
I20250110 20:51:47.882705 2425 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=7, attempt_no=0}) took 1116 ms. Trace:
I20250110 20:51:47.882848 2425 rpcz_store.cc:276] 0110 20:51:46.766072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:46.766140 (+ 68us) service_pool.cc:225] Handling call
0110 20:51:47.882637 (+1116497us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:47.211743 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:47.211865 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:47.211870 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:47.211873 (+ 3us) tablet.cc:659] Decoding operations
0110 20:51:47.211966 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:47.211978 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:47.211983 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:47.212034 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:51:47.212036 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:47.212073 (+ 37us) write_op.cc:270] Start()
0110 20:51:47.882202 (+670129us) write_op.cc:276] Timestamp: P: 1736542307212027 usec, L: 0
0110 20:51:47.882208 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:47.882256 (+ 48us) op_driver.cc:389] HandleFailure(Aborted: Op submitted in term 1 cannot be replicated in term 3)
0110 20:51:47.882275 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:47.882292 (+ 17us) write_op.cc:454] Released schema lock
0110 20:51:47.882300 (+ 8us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":445129}]]}
W20250110 20:51:47.886202 2036 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:46820: Illegal state: replica 9ea09a0a2f2549a4a06723c0478f03f1 is not leader of this config: current role FOLLOWER
W20250110 20:51:47.892153 2167 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56042: Illegal state: replica 555db90ad09d47708f85e4658c394115 is not leader of this config: current role FOLLOWER
I20250110 20:51:49.518167 2474 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:49.518427 2474 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:49.519411 2474 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:49.519886 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:49.520084 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:49.520768 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:49.521380 2479 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=6, attempt_no=3}) took 2271 ms. Trace:
I20250110 20:51:49.521497 2479 rpcz_store.cc:276] 0110 20:51:47.250018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:47.250082 (+ 64us) service_pool.cc:225] Handling call
0110 20:51:49.521346 (+2271264us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:47.883446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:47.883547 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:47.883552 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:47.883554 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:47.883653 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:47.883665 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:47.883669 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:47.883719 (+ 50us) tablet.cc:698] Row locks acquired
0110 20:51:47.883721 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:47.883770 (+ 49us) write_op.cc:270] Start()
0110 20:51:48.642914 (+759144us) write_op.cc:276] Timestamp: P: 1736542307883709 usec, L: 0
0110 20:51:48.642920 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:48.643137 (+ 217us) log.cc:844] Serialized 354 byte log entry
0110 20:51:49.520275 (+877138us) op_driver.cc:464] REPLICATION: finished
0110 20:51:49.520745 (+ 470us) write_op.cc:301] APPLY: starting
0110 20:51:49.520765 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:49.520809 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:49.520838 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:49.520963 (+ 125us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:49.520997 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:49.521004 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:49.521132 (+ 128us) log.cc:844] Serialized 22 byte log entry
0110 20:51:49.521229 (+ 97us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:49.521250 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:49.521257 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":400,"num_ops":1,"prepare.queue_time_us":632945,"prepare.run_cpu_time_us":1043,"prepare.run_wall_time_us":760105,"replication_time_us":877321,"spinlock_wait_cycles":247040,"thread_start_us":1131,"threads_started":2,"wal-append.queue_time_us":400}]]}
I20250110 20:51:49.637862 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:49.638098 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:49.638974 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:49.639436 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:49.639467 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 9 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:49.640038 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:49.642174 2474 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:49.643601 2479 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=7, attempt_no=3}) took 1741 ms. Trace:
I20250110 20:51:49.643716 2479 rpcz_store.cc:276] 0110 20:51:47.902154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:47.902210 (+ 56us) service_pool.cc:225] Handling call
0110 20:51:49.643533 (+1741323us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:48.643600 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:48.643734 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:48.643741 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:51:48.643744 (+ 3us) tablet.cc:659] Decoding operations
0110 20:51:48.643845 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:48.643858 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:48.643862 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:48.643946 (+ 84us) tablet.cc:698] Row locks acquired
0110 20:51:48.643949 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:51:48.644000 (+ 51us) write_op.cc:270] Start()
0110 20:51:48.716152 (+ 72152us) write_op.cc:276] Timestamp: P: 1736542308643951 usec, L: 0
0110 20:51:48.716158 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:48.716401 (+ 243us) log.cc:844] Serialized 354 byte log entry
0110 20:51:49.642952 (+926551us) op_driver.cc:464] REPLICATION: finished
0110 20:51:49.643066 (+ 114us) write_op.cc:301] APPLY: starting
0110 20:51:49.643085 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:49.643126 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:49.643162 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:49.643248 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:49.643276 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:49.643281 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:51:49.643350 (+ 69us) log.cc:844] Serialized 22 byte log entry
0110 20:51:49.643424 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:49.643441 (+ 17us) write_op.cc:454] Released schema lock
0110 20:51:49.643448 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":740887,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":72881,"replication_time_us":926743}]]}
W20250110 20:51:49.937430 2036 tablet_service.cc:2970] error setting up scanner e656177eb86743bf9c7c4283eb2d0f0b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.294s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:50.226361 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:50.986640 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:50.986888 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:50.987680 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:50.988175 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:50.988183 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:50.988942 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:50.989482 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:50.990057 2491 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=8, attempt_no=0}) took 1466 ms. Trace:
I20250110 20:51:50.990150 2491 rpcz_store.cc:276] 0110 20:51:49.523293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:49.523364 (+ 71us) service_pool.cc:225] Handling call
0110 20:51:50.990017 (+1466653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:49.524123 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:49.524267 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:49.524272 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:49.524273 (+ 1us) tablet.cc:659] Decoding operations
0110 20:51:49.524354 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:49.524362 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:49.524366 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:49.524424 (+ 58us) tablet.cc:698] Row locks acquired
0110 20:51:49.524426 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:49.524483 (+ 57us) write_op.cc:270] Start()
0110 20:51:50.404629 (+880146us) write_op.cc:276] Timestamp: P: 1736542309524407 usec, L: 0
0110 20:51:50.404633 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:51:50.404888 (+ 255us) log.cc:844] Serialized 354 byte log entry
0110 20:51:50.988136 (+583248us) op_driver.cc:464] REPLICATION: finished
0110 20:51:50.989067 (+ 931us) write_op.cc:301] APPLY: starting
0110 20:51:50.989089 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:50.989139 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:50.989174 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:50.989283 (+ 109us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:50.989319 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:50.989325 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:51:50.989445 (+ 120us) log.cc:844] Serialized 22 byte log entry
0110 20:51:50.989915 (+ 470us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:50.989930 (+ 15us) write_op.cc:454] Released schema lock
0110 20:51:50.989935 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":832,"num_ops":1,"prepare.queue_time_us":321,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":880864,"replication_time_us":583451,"thread_start_us":950,"threads_started":2}]]}
I20250110 20:51:51.197973 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:51.198192 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:51.199096 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:51.199575 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:51.200232 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 11 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:51.200714 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
W20250110 20:51:51.474944 2036 tablet_service.cc:2970] error setting up scanner 773b9eb6fdcd48a6bdcfad5b808cc198 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.275s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
I20250110 20:51:51.560045 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:51.560262 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:51.561098 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:51.561547 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:51.561817 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 12 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:51.562325 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:51.562517 2498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=9, attempt_no=0}) took 1917 ms. Trace:
I20250110 20:51:51.562662 2498 rpcz_store.cc:276] 0110 20:51:49.645526 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:49.645586 (+ 60us) service_pool.cc:225] Handling call
0110 20:51:51.562469 (+1916883us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:50.405029 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:50.405179 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:50.405184 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:50.405186 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:50.405275 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:50.405286 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:50.405290 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:50.405349 (+ 59us) tablet.cc:698] Row locks acquired
0110 20:51:50.405352 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:51:50.405408 (+ 56us) write_op.cc:270] Start()
0110 20:51:51.051548 (+646140us) write_op.cc:276] Timestamp: P: 1736542310405346 usec, L: 0
0110 20:51:51.051552 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:51:51.051767 (+ 215us) log.cc:844] Serialized 354 byte log entry
0110 20:51:51.561179 (+509412us) op_driver.cc:464] REPLICATION: finished
0110 20:51:51.561689 (+ 510us) write_op.cc:301] APPLY: starting
0110 20:51:51.561721 (+ 32us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:51.561765 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:51.561801 (+ 36us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:51.561881 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:51.561922 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:51.561929 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:51.562081 (+ 152us) log.cc:844] Serialized 22 byte log entry
0110 20:51:51.562340 (+ 259us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:51.562361 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:51.562368 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":450,"num_ops":1,"prepare.queue_time_us":758994,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":646824,"replication_time_us":509589,"thread_start_us":371,"threads_started":1}]]}
I20250110 20:51:51.687614 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:52.145545 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:52.145745 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:52.146526 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:52.147351 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:52.147583 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 13 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:52.148451 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:52.148459 2501 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=10, attempt_no=0}) took 1156 ms. Trace:
I20250110 20:51:52.148594 2501 rpcz_store.cc:276] 0110 20:51:50.991940 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:50.991995 (+ 55us) service_pool.cc:225] Handling call
0110 20:51:52.148397 (+1156402us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:51.051894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:51.052020 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:51.052025 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:51.052027 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:51.052126 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:51.052138 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:51.052143 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:51.052216 (+ 73us) tablet.cc:698] Row locks acquired
0110 20:51:51.052218 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:51.052280 (+ 62us) write_op.cc:270] Start()
0110 20:51:51.415497 (+363217us) write_op.cc:276] Timestamp: P: 1736542311052220 usec, L: 0
0110 20:51:51.415503 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:51.415735 (+ 232us) log.cc:844] Serialized 354 byte log entry
0110 20:51:52.147097 (+731362us) op_driver.cc:464] REPLICATION: finished
0110 20:51:52.147911 (+ 814us) write_op.cc:301] APPLY: starting
0110 20:51:52.147931 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:52.147976 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:52.148001 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:52.148070 (+ 69us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:52.148104 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:52.148111 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:52.148195 (+ 84us) log.cc:844] Serialized 22 byte log entry
0110 20:51:52.148268 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:52.148289 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:52.148297 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":742,"num_ops":1,"prepare.queue_time_us":59413,"prepare.run_cpu_time_us":810,"prepare.run_wall_time_us":363918,"replication_time_us":731557,"thread_start_us":794,"threads_started":1}]]}
I20250110 20:51:52.402031 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:52.445142 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:53.024752 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:53.024994 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:53.025734 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:53.026237 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:53.026255 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 15 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:53.026881 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:53.027258 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:53.082346 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:53.082528 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:53.083412 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:53.084123 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:53.084143 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 14 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:53.084735 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:53.693181 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:53.693377 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:53.694242 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:53.694710 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:53.694916 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 16 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:53.695581 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:53.695933 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:53.696446 2514 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=13, attempt_no=0}) took 1291 ms. Trace:
I20250110 20:51:53.696637 2514 rpcz_store.cc:276] 0110 20:51:52.405144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:52.405212 (+ 68us) service_pool.cc:225] Handling call
0110 20:51:53.696390 (+1291178us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:52.493927 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:52.494067 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:52.494075 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:51:52.494078 (+ 3us) tablet.cc:659] Decoding operations
0110 20:51:52.494193 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:52.494205 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:52.494211 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:52.494280 (+ 69us) tablet.cc:698] Row locks acquired
0110 20:51:52.494282 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:52.494336 (+ 54us) write_op.cc:270] Start()
0110 20:51:52.839517 (+345181us) write_op.cc:276] Timestamp: P: 1736542312494276 usec, L: 0
0110 20:51:52.839523 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:52.839772 (+ 249us) log.cc:844] Serialized 354 byte log entry
0110 20:51:53.695487 (+855715us) op_driver.cc:464] REPLICATION: finished
0110 20:51:53.695859 (+ 372us) write_op.cc:301] APPLY: starting
0110 20:51:53.695877 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:53.695918 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:53.695947 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:53.696018 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:53.696052 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:53.696058 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:51:53.696141 (+ 83us) log.cc:844] Serialized 22 byte log entry
0110 20:51:53.696252 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:53.696271 (+ 19us) write_op.cc:454] Released schema lock
0110 20:51:53.696278 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":304,"num_ops":1,"prepare.queue_time_us":88296,"prepare.run_cpu_time_us":843,"prepare.run_wall_time_us":345921,"replication_time_us":855930,"thread_start_us":235,"threads_started":1}]]}
I20250110 20:51:54.108587 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:54.108785 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:54.109614 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:54.110316 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:54.110333 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 17 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:54.110460 2514 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=14, attempt_no=0}) took 1081 ms. Trace:
I20250110 20:51:54.110550 2514 rpcz_store.cc:276] 0110 20:51:53.029269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:53.029369 (+ 100us) service_pool.cc:225] Handling call
0110 20:51:54.110411 (+1081042us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:53.029829 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:53.029934 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:53.029938 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:51:53.029940 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:53.030010 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:53.030018 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:53.030044 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:53.030092 (+ 48us) tablet.cc:698] Row locks acquired
0110 20:51:53.030094 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:53.030133 (+ 39us) write_op.cc:270] Start()
0110 20:51:53.938287 (+908154us) write_op.cc:276] Timestamp: P: 1736542313030063 usec, L: 0
0110 20:51:53.938293 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:53.938528 (+ 235us) log.cc:844] Serialized 354 byte log entry
0110 20:51:54.109579 (+171051us) op_driver.cc:464] REPLICATION: finished
0110 20:51:54.109707 (+ 128us) write_op.cc:301] APPLY: starting
0110 20:51:54.109730 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:54.109777 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:54.109809 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:54.110022 (+ 213us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:54.110060 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:54.110067 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:54.110173 (+ 106us) log.cc:844] Serialized 22 byte log entry
0110 20:51:54.110283 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:54.110304 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:54.110315 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":699,"prepare.run_wall_time_us":908776,"replication_time_us":171250}]]}
I20250110 20:51:54.111594 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:54.443899 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:54.444126 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:34181 (request call id 1912) took 1360 ms. Trace:
I20250110 20:51:54.444332 2206 rpcz_store.cc:276] 0110 20:51:53.083982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:53.084037 (+ 55us) service_pool.cc:225] Handling call
0110 20:51:53.084250 (+ 213us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:51:53.084400 (+ 150us) raft_consensus.cc:1529] Early marking committed up to index 0
0110 20:51:53.084405 (+ 5us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:51:53.084534 (+ 129us) write_op.cc:270] Start()
0110 20:51:53.783679 (+699145us) write_op.cc:276] Timestamp: P: 1736542312151345 usec, L: 0
0110 20:51:53.784285 (+ 606us) write_op.cc:270] Start()
0110 20:51:54.442489 (+658204us) write_op.cc:276] Timestamp: P: 1736542312494276 usec, L: 0
0110 20:51:54.443149 (+ 660us) log.cc:844] Serialized 708 byte log entry
0110 20:51:54.443448 (+ 299us) raft_consensus.cc:1640] Marking committed up to 15
0110 20:51:54.443772 (+ 324us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:51:54.443814 (+ 42us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:51:54.443987 (+ 173us) raft_consensus.cc:1680] finished
0110 20:51:54.443990 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:51:54.444056 (+ 66us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:53.784169 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:53.784304 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:53.784312 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:51:53.784314 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:53.784430 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:53.784450 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:53.784455 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:53.784512 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:51:53.784514 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:54.443498 (+658984us) op_driver.cc:464] REPLICATION: finished
0110 20:51:54.443873 (+ 375us) write_op.cc:301] APPLY: starting
0110 20:51:54.443895 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:54.443937 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:54.443939 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:54.444008 (+ 69us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:54.444042 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:54.444108 (+ 66us) write_op.cc:312] APPLY: finished
0110 20:51:54.444197 (+ 89us) log.cc:844] Serialized 22 byte log entry
0110 20:51:54.444282 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:54.444382 (+ 100us) write_op.cc:454] Released schema lock
0110 20:51:54.444390 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 20:51:54.444500 (+ 110us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:54.444504 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:51:54.443031 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:54.443204 (+ 173us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:54.443212 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:51:54.443214 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:54.443318 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:54.443365 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:54.443370 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:54.443438 (+ 68us) tablet.cc:698] Row locks acquired
0110 20:51:54.443440 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":223,"threads_started":1,"wal-append.queue_time_us":299,"child_traces":[["op",{"apply.queue_time_us":298,"apply.run_cpu_time_us":565,"apply.run_wall_time_us":625,"num_ops":1,"partition_lock_wait_us":9,"prepare.queue_time_us":345,"prepare.run_cpu_time_us":452,"prepare.run_wall_time_us":451,"replication_time_us":1358941,"thread_start_us":447,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":35,"prepare.queue_time_us":383,"prepare.run_cpu_time_us":498,"prepare.run_wall_time_us":499,"thread_start_us":250,"threads_started":1}]]}
W20250110 20:51:54.804366 2036 tablet_service.cc:2970] error setting up scanner 48d521e881ef43e1809fcdd12d0e5ab2 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.694s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:54.939764 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:54.941943 2522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=15, attempt_no=0}) took 1243 ms. Trace:
I20250110 20:51:54.942108 2522 rpcz_store.cc:276] 0110 20:51:53.698449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:53.698519 (+ 70us) service_pool.cc:225] Handling call
0110 20:51:54.941884 (+1243365us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:53.938648 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:53.938804 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:53.938809 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:53.938811 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:53.938896 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:53.938909 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:53.938912 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:53.938967 (+ 55us) tablet.cc:698] Row locks acquired
0110 20:51:53.938969 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:53.939019 (+ 50us) write_op.cc:270] Start()
0110 20:51:53.999177 (+ 60158us) write_op.cc:276] Timestamp: P: 1736542313938975 usec, L: 0
0110 20:51:53.999182 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:51:53.999445 (+ 263us) log.cc:844] Serialized 354 byte log entry
0110 20:51:54.940941 (+941496us) op_driver.cc:464] REPLICATION: finished
0110 20:51:54.941307 (+ 366us) write_op.cc:301] APPLY: starting
0110 20:51:54.941325 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:54.941361 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:54.941408 (+ 47us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:54.941512 (+ 104us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:54.941548 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:54.941554 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:51:54.941646 (+ 92us) log.cc:844] Serialized 22 byte log entry
0110 20:51:54.941733 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:54.941754 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:54.941761 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":310,"num_ops":1,"prepare.queue_time_us":239618,"prepare.run_cpu_time_us":814,"prepare.run_wall_time_us":60875,"replication_time_us":941726,"thread_start_us":238,"threads_started":1}]]}
I20250110 20:51:55.813340 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:55.813567 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:55.814368 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:55.814833 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:55.815094 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 19 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:55.815732 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:55.816040 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:55.816586 2528 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=16, attempt_no=0}) took 1704 ms. Trace:
I20250110 20:51:55.816709 2528 rpcz_store.cc:276] 0110 20:51:54.112288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:54.112361 (+ 73us) service_pool.cc:225] Handling call
0110 20:51:55.816530 (+1704169us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:54.112804 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:54.112900 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:54.112904 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:51:54.112906 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:54.112977 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:54.112986 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:54.112989 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:54.113027 (+ 38us) tablet.cc:698] Row locks acquired
0110 20:51:54.113029 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:54.113061 (+ 32us) write_op.cc:270] Start()
0110 20:51:54.798232 (+685171us) write_op.cc:276] Timestamp: P: 1736542314113003 usec, L: 0
0110 20:51:54.798238 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:54.798500 (+ 262us) log.cc:844] Serialized 354 byte log entry
0110 20:51:55.815505 (+1017005us) op_driver.cc:464] REPLICATION: finished
0110 20:51:55.815970 (+ 465us) write_op.cc:301] APPLY: starting
0110 20:51:55.815990 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:55.816035 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:55.816061 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:55.816135 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:55.816170 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:55.816177 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:55.816261 (+ 84us) log.cc:844] Serialized 22 byte log entry
0110 20:51:55.816359 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:55.816381 (+ 22us) write_op.cc:454] Released schema lock
0110 20:51:55.816389 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":374,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":694,"prepare.run_wall_time_us":685768,"replication_time_us":1017233,"thread_start_us":295,"threads_started":1}]]}
I20250110 20:51:56.037824 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:56.038010 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:34181 (request call id 1913) took 1591 ms. Trace:
I20250110 20:51:56.038038 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:56.038179 2206 rpcz_store.cc:276] 0110 20:51:54.446566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:54.446626 (+ 60us) service_pool.cc:225] Handling call
0110 20:51:54.446830 (+ 204us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:51:54.446967 (+ 137us) raft_consensus.cc:1529] Early marking committed up to index 16
0110 20:51:54.447055 (+ 88us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:51:54.447169 (+ 114us) write_op.cc:270] Start()
0110 20:51:55.226321 (+779152us) write_op.cc:276] Timestamp: P: 1736542313030063 usec, L: 0
0110 20:51:55.227047 (+ 726us) write_op.cc:270] Start()
0110 20:51:56.036253 (+809206us) write_op.cc:276] Timestamp: P: 1736542313938975 usec, L: 0
0110 20:51:56.036998 (+ 745us) log.cc:844] Serialized 708 byte log entry
0110 20:51:56.037347 (+ 349us) raft_consensus.cc:1640] Marking committed up to 17
0110 20:51:56.037695 (+ 348us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:51:56.037733 (+ 38us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:51:56.037891 (+ 158us) raft_consensus.cc:1680] finished
0110 20:51:56.037894 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:51:56.037954 (+ 60us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:55.226913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:55.227056 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:55.227062 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:51:55.227064 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:55.227159 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:55.227173 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:55.227176 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:55.227224 (+ 48us) tablet.cc:698] Row locks acquired
0110 20:51:55.227226 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:56.037397 (+810171us) op_driver.cc:464] REPLICATION: finished
0110 20:51:56.037820 (+ 423us) write_op.cc:301] APPLY: starting
0110 20:51:56.037839 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:56.037882 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:56.037884 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:56.038145 (+ 261us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:56.038184 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:56.038214 (+ 30us) write_op.cc:312] APPLY: finished
Related trace 'op':
0110 20:51:56.036817 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:56.036939 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:56.036945 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:51:56.036946 (+ 1us) tablet.cc:659] Decoding operations
0110 20:51:56.037027 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:56.037041 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:56.037045 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:56.037095 (+ 50us) tablet.cc:698] Row locks acquired
0110 20:51:56.037097 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":252,"threads_started":1,"wal-append.queue_time_us":326,"child_traces":[["op",{"apply.queue_time_us":319,"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":445,"prepare.run_cpu_time_us":412,"prepare.run_wall_time_us":412,"replication_time_us":1590206,"thread_start_us":503,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":411,"prepare.run_cpu_time_us":372,"prepare.run_wall_time_us":371,"thread_start_us":245,"threads_started":1}]]}
I20250110 20:51:56.039058 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:56.039578 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:56.039716 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 18 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:56.040378 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
W20250110 20:51:56.085901 2036 tablet_service.cc:2970] error setting up scanner 60de613e29824431b1b8706c84130afe with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.269s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
I20250110 20:51:56.097636 2487 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:56.097844 2487 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:56.098613 2487 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:56.099192 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:56.099408 2528 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=17, attempt_no=0}) took 1155 ms. Trace:
I20250110 20:51:56.099367 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 20 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:56.099521 2528 rpcz_store.cc:276] 0110 20:51:54.943890 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:54.943962 (+ 72us) service_pool.cc:225] Handling call
0110 20:51:56.099344 (+1155382us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:54.944506 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:54.944605 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:54.944610 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:54.944611 (+ 1us) tablet.cc:659] Decoding operations
0110 20:51:54.944685 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:54.944693 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:54.944697 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:54.944742 (+ 45us) tablet.cc:698] Row locks acquired
0110 20:51:54.944744 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:54.944786 (+ 42us) write_op.cc:270] Start()
0110 20:51:55.460939 (+516153us) write_op.cc:276] Timestamp: P: 1736542314944742 usec, L: 0
0110 20:51:55.460944 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:51:55.461189 (+ 245us) log.cc:844] Serialized 354 byte log entry
0110 20:51:56.098636 (+637447us) op_driver.cc:464] REPLICATION: finished
0110 20:51:56.098800 (+ 164us) write_op.cc:301] APPLY: starting
0110 20:51:56.098820 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:56.098865 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:56.098903 (+ 38us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:56.098979 (+ 76us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:56.099031 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:56.099045 (+ 14us) write_op.cc:312] APPLY: finished
0110 20:51:56.099138 (+ 93us) log.cc:844] Serialized 22 byte log entry
0110 20:51:56.099219 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:56.099240 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:56.099248 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":105,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":708,"prepare.run_wall_time_us":516764,"replication_time_us":637654}]]}
I20250110 20:51:56.099958 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:56.168949 2487 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:57.117107 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:57.117448 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:34181 (request call id 1914) took 1077 ms. Trace:
I20250110 20:51:57.117588 2206 rpcz_store.cc:276] 0110 20:51:56.040116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:56.040303 (+ 187us) service_pool.cc:225] Handling call
0110 20:51:56.040523 (+ 220us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:51:56.040662 (+ 139us) raft_consensus.cc:1529] Early marking committed up to index 18
0110 20:51:56.040757 (+ 95us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:51:56.040880 (+ 123us) write_op.cc:270] Start()
0110 20:51:56.155070 (+114190us) write_op.cc:276] Timestamp: P: 1736542314113003 usec, L: 0
0110 20:51:56.155453 (+ 383us) write_op.cc:270] Start()
0110 20:51:57.115633 (+960180us) write_op.cc:276] Timestamp: P: 1736542314944742 usec, L: 0
0110 20:51:57.116294 (+ 661us) log.cc:844] Serialized 708 byte log entry
0110 20:51:57.116625 (+ 331us) raft_consensus.cc:1640] Marking committed up to 19
0110 20:51:57.116956 (+ 331us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:51:57.117000 (+ 44us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:51:57.117267 (+ 267us) raft_consensus.cc:1680] finished
0110 20:51:57.117270 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:51:57.117331 (+ 61us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:56.155322 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:56.155436 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:56.155442 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:51:56.155444 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:56.155525 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:56.155543 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:56.155548 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:56.155613 (+ 65us) tablet.cc:698] Row locks acquired
0110 20:51:56.155615 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:57.116673 (+961058us) op_driver.cc:464] REPLICATION: finished
0110 20:51:57.117154 (+ 481us) write_op.cc:301] APPLY: starting
0110 20:51:57.117172 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:57.117217 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:57.117220 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:57.117291 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:57.117345 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:57.117462 (+ 117us) write_op.cc:312] APPLY: finished
0110 20:51:57.117545 (+ 83us) log.cc:844] Serialized 22 byte log entry
0110 20:51:57.117611 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:57.117627 (+ 16us) write_op.cc:454] Released schema lock
0110 20:51:57.117634 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 20:51:57.117732 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
Related trace 'op':
0110 20:51:57.116183 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:57.116293 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:57.116299 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:51:57.116301 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:57.116389 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:57.116402 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:57.116405 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:57.116462 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:51:57.116463 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":229,"threads_started":1,"wal-append.queue_time_us":332,"child_traces":[["op",{"apply.queue_time_us":335,"apply.run_cpu_time_us":594,"apply.run_wall_time_us":605,"num_ops":1,"partition_lock_wait_us":9,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":407,"prepare.run_wall_time_us":408,"replication_time_us":1075771,"spinlock_wait_cycles":19328,"thread_start_us":217,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":400,"prepare.run_cpu_time_us":395,"prepare.run_wall_time_us":395,"thread_start_us":218,"threads_started":1}]]}
I20250110 20:51:57.892606 2533 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:57.892802 2533 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:57.893879 2533 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:57.894405 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:57.895392 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:57.895946 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:57.896363 2542 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=19, attempt_no=0}) took 1795 ms. Trace:
I20250110 20:51:57.896481 2542 rpcz_store.cc:276] 0110 20:51:56.101329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:56.101408 (+ 79us) service_pool.cc:225] Handling call
0110 20:51:57.896290 (+1794882us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:56.101911 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:56.102009 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:56.102014 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:51:56.102016 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:56.102142 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:56.102152 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:56.102156 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:56.102199 (+ 43us) tablet.cc:698] Row locks acquired
0110 20:51:56.102200 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:51:56.102235 (+ 35us) write_op.cc:270] Start()
0110 20:51:56.909364 (+807129us) write_op.cc:276] Timestamp: P: 1736542316102181 usec, L: 0
0110 20:51:56.909370 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:56.909586 (+ 216us) log.cc:844] Serialized 354 byte log entry
0110 20:51:57.894547 (+984961us) op_driver.cc:464] REPLICATION: finished
0110 20:51:57.895475 (+ 928us) write_op.cc:301] APPLY: starting
0110 20:51:57.895494 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:57.895537 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:57.895568 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:57.895664 (+ 96us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:57.895705 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:57.895714 (+ 9us) write_op.cc:312] APPLY: finished
0110 20:51:57.895939 (+ 225us) log.cc:844] Serialized 22 byte log entry
0110 20:51:57.896051 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:57.896072 (+ 21us) write_op.cc:454] Released schema lock
0110 20:51:57.896079 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":842,"num_ops":1,"prepare.queue_time_us":75,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":807753,"replication_time_us":985125,"thread_start_us":685,"threads_started":1}]]}
I20250110 20:51:58.583096 2533 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:58.586352 2548 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=20, attempt_no=0}) took 2414 ms. Trace:
I20250110 20:51:58.586503 2548 rpcz_store.cc:276] 0110 20:51:56.172093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:56.172174 (+ 81us) service_pool.cc:225] Handling call
0110 20:51:58.586288 (+2414114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:56.909709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:56.909831 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:56.909838 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:51:56.909841 (+ 3us) tablet.cc:659] Decoding operations
0110 20:51:56.909940 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:56.909951 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:56.909956 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:56.910015 (+ 59us) tablet.cc:698] Row locks acquired
0110 20:51:56.910017 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:56.910062 (+ 45us) write_op.cc:270] Start()
0110 20:51:57.234251 (+324189us) write_op.cc:276] Timestamp: P: 1736542316910018 usec, L: 0
0110 20:51:57.234257 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:57.234515 (+ 258us) log.cc:844] Serialized 354 byte log entry
0110 20:51:58.585266 (+1350751us) op_driver.cc:464] REPLICATION: finished
0110 20:51:58.585701 (+ 435us) write_op.cc:301] APPLY: starting
0110 20:51:58.585721 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:58.585773 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:58.585828 (+ 55us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:58.585909 (+ 81us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:58.585944 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:58.585950 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:51:58.586046 (+ 96us) log.cc:844] Serialized 22 byte log entry
0110 20:51:58.586139 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:58.586158 (+ 19us) write_op.cc:454] Released schema lock
0110 20:51:58.586166 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":337,"num_ops":1,"prepare.queue_time_us":737033,"prepare.run_cpu_time_us":806,"prepare.run_wall_time_us":324887,"replication_time_us":1350948,"thread_start_us":257,"threads_started":1}]]}
I20250110 20:51:58.597581 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:58.597653 2206 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:34181 (request call id 1915) took 1478 ms. Trace:
I20250110 20:51:58.597785 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:58.597810 2206 rpcz_store.cc:276] 0110 20:51:57.119355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:57.119427 (+ 72us) service_pool.cc:225] Handling call
0110 20:51:57.119626 (+ 199us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:51:57.119750 (+ 124us) raft_consensus.cc:1529] Early marking committed up to index 20
0110 20:51:57.119829 (+ 79us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:51:57.119949 (+ 120us) write_op.cc:270] Start()
0110 20:51:57.660139 (+540190us) write_op.cc:276] Timestamp: P: 1736542315819277 usec, L: 0
0110 20:51:57.660754 (+ 615us) write_op.cc:270] Start()
0110 20:51:58.595949 (+935195us) write_op.cc:276] Timestamp: P: 1736542316102181 usec, L: 0
0110 20:51:58.596687 (+ 738us) log.cc:844] Serialized 708 byte log entry
0110 20:51:58.597004 (+ 317us) raft_consensus.cc:1640] Marking committed up to 21
0110 20:51:58.597465 (+ 461us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:51:58.597508 (+ 43us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:51:58.597519 (+ 11us) raft_consensus.cc:1680] finished
0110 20:51:58.597522 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:51:58.597596 (+ 74us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:57.660686 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:57.660793 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:57.660799 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:51:57.660801 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:57.660880 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:57.660894 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:57.660897 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:57.660951 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:51:57.660953 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:58.597056 (+936103us) op_driver.cc:464] REPLICATION: finished
0110 20:51:58.597606 (+ 550us) write_op.cc:301] APPLY: starting
0110 20:51:58.597624 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:58.597673 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:58.597676 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:58.597747 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:58.597784 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:58.597823 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:51:58.597913 (+ 90us) log.cc:844] Serialized 22 byte log entry
Related trace 'op':
0110 20:51:58.596509 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:58.596616 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:58.596622 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:51:58.596623 (+ 1us) tablet.cc:659] Decoding operations
0110 20:51:58.596703 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:58.596715 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:58.596718 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:58.596764 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:51:58.596766 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":230,"threads_started":1,"wal-append.queue_time_us":317,"child_traces":[["op",{"apply.queue_time_us":469,"apply.run_cpu_time_us":523,"apply.run_wall_time_us":522,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":396,"prepare.run_cpu_time_us":405,"prepare.run_wall_time_us":405,"replication_time_us":1477082,"spinlock_wait_cycles":4480,"thread_start_us":600,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":377,"prepare.run_cpu_time_us":350,"prepare.run_wall_time_us":349,"thread_start_us":256,"threads_started":1}]]}
I20250110 20:51:58.598862 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:58.600503 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:58.600776 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 22 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:58.601403 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:58.672091 2533 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:58.672298 2533 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:58.673185 2533 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:58.673933 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:58.675241 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 24 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:58.675832 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:58.676131 2533 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:59.023286 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:51:59.161027 2036 tablet_service.cc:2970] error setting up scanner 67598ad8ab8849a2b8d5828bc2492824 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.487s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
W20250110 20:51:59.415278 2036 tablet_service.cc:2970] error setting up scanner 4c66398567e34c5e86942a520e5611dc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.741s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "d83f94bd4eb34a37aad2beaffbb1f42b"
I20250110 20:51:59.592804 2473 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:59.593006 2473 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:59.593793 2473 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:59.594269 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:51:59.594429 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:59.595032 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:51:59.595496 2556 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=22, attempt_no=0}) took 1006 ms. Trace:
I20250110 20:51:59.595567 2473 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:51:59.595659 2556 rpcz_store.cc:276] 0110 20:51:58.588611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:58.588683 (+ 72us) service_pool.cc:225] Handling call
0110 20:51:59.595450 (+1006767us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:58.589144 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:58.589322 (+ 178us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:58.589330 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:51:58.589332 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:58.589426 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:58.589438 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:58.589443 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:58.589494 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:51:58.589497 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:51:58.589540 (+ 43us) write_op.cc:270] Start()
0110 20:51:59.174689 (+585149us) write_op.cc:276] Timestamp: P: 1736542318589481 usec, L: 0
0110 20:51:59.174695 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:51:59.174934 (+ 239us) log.cc:844] Serialized 354 byte log entry
0110 20:51:59.593813 (+418879us) op_driver.cc:464] REPLICATION: finished
0110 20:51:59.594527 (+ 714us) write_op.cc:301] APPLY: starting
0110 20:51:59.594545 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:51:59.594594 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 20:51:59.594620 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:51:59.594694 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:51:59.594727 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:51:59.594734 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:51:59.594868 (+ 134us) log.cc:844] Serialized 22 byte log entry
0110 20:51:59.595151 (+ 283us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:51:59.595171 (+ 20us) write_op.cc:454] Released schema lock
0110 20:51:59.595178 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":649,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":585893,"replication_time_us":419079,"thread_start_us":561,"threads_started":1}]]}
I20250110 20:51:59.740522 2554 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:51:59.740725 2554 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:51:59.741564 2554 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:51:59.742074 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:51:59.742094 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:51:59.742630 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:51:59.743141 2554 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:00.007791 2554 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:00.007958 2554 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:00.008607 2554 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:00.009088 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:00.009694 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:00.010175 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:00.010434 2554 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:00.077776 2554 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:00.077951 2554 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:00.078565 2554 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:00.079130 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:00.079556 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 25 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:00.080032 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:00.080300 2554 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:00.157002 2565 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:00.157238 2565 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:00.158486 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:00.158675 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:00.159327 2567 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=23, attempt_no=0}) took 1483 ms. Trace:
I20250110 20:52:00.159385 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:00.159612 2565 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:00.159447 2567 rpcz_store.cc:276] 0110 20:51:58.676051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:58.676101 (+ 50us) service_pool.cc:225] Handling call
0110 20:52:00.159270 (+1483169us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:51:59.175072 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:51:59.175205 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:51:59.175213 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:51:59.175215 (+ 2us) tablet.cc:659] Decoding operations
0110 20:51:59.175317 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:51:59.175329 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:51:59.175334 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:51:59.175388 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:51:59.175390 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:51:59.175439 (+ 49us) write_op.cc:270] Start()
0110 20:52:00.133590 (+958151us) write_op.cc:276] Timestamp: P: 1736542319175363 usec, L: 0
0110 20:52:00.133594 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:52:00.133798 (+ 204us) log.cc:844] Serialized 354 byte log entry
0110 20:52:00.157946 (+ 24148us) op_driver.cc:464] REPLICATION: finished
0110 20:52:00.158637 (+ 691us) write_op.cc:301] APPLY: starting
0110 20:52:00.158657 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:00.158707 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:00.158766 (+ 59us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:00.158843 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:00.158879 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:00.158886 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:00.159037 (+ 151us) log.cc:844] Serialized 22 byte log entry
0110 20:52:00.159128 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:00.159150 (+ 22us) write_op.cc:454] Released schema lock
0110 20:52:00.159158 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":612,"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":498483,"prepare.run_cpu_time_us":789,"prepare.run_wall_time_us":958859,"raft.queue_time_us":70,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":326,"replication_time_us":24313,"spinlock_wait_cycles":200448,"thread_start_us":500,"threads_started":1}]]}
I20250110 20:52:00.160426 2565 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:00.545322 2554 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:00.545547 2554 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:00.546389 2554 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:00.546586 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:00.547163 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 26 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:00.547698 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:00.555142 2554 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:00.969120 2554 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:00.969331 2554 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:00.970124 2554 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:00.970641 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:00.971645 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:00.971809 2572 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=24, attempt_no=0}) took 1374 ms. Trace:
I20250110 20:52:00.971920 2572 rpcz_store.cc:276] 0110 20:51:59.597509 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:51:59.597590 (+ 81us) service_pool.cc:225] Handling call
0110 20:52:00.971758 (+1374168us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:00.133970 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:00.134098 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:00.134106 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:00.134109 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:00.134210 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:00.134221 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:00.134226 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:00.134283 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:52:00.134286 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:00.134339 (+ 53us) write_op.cc:270] Start()
0110 20:52:00.622491 (+488152us) write_op.cc:276] Timestamp: P: 1736542320134283 usec, L: 0
0110 20:52:00.622496 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:00.622707 (+ 211us) log.cc:844] Serialized 354 byte log entry
0110 20:52:00.970214 (+347507us) op_driver.cc:464] REPLICATION: finished
0110 20:52:00.971161 (+ 947us) write_op.cc:301] APPLY: starting
0110 20:52:00.971180 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:00.971232 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:00.971289 (+ 57us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:00.971400 (+ 111us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:00.971438 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:00.971444 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:00.971551 (+ 107us) log.cc:844] Serialized 22 byte log entry
0110 20:52:00.971630 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:00.971649 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:00.971656 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":883,"num_ops":1,"prepare.queue_time_us":535824,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":488853,"replication_time_us":347667,"thread_start_us":786,"threads_started":1}]]}
I20250110 20:52:00.972394 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:00.972779 2554 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:01.158423 2569 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:01.158615 2569 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:01.159505 2569 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:01.160159 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:01.160526 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 27 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:01.161118 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:01.161378 2569 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:01.345386 2554 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:01.345610 2554 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:01.346411 2554 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:01.346988 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:01.346983 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:01.347311 2572 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=25, attempt_no=0}) took 1185 ms. Trace:
I20250110 20:52:01.347429 2572 rpcz_store.cc:276] 0110 20:52:00.161676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:00.161727 (+ 51us) service_pool.cc:225] Handling call
0110 20:52:01.347248 (+1185521us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:00.622866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:00.623002 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:00.623010 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:00.623013 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:00.623113 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:00.623125 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:00.623129 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:00.623196 (+ 67us) tablet.cc:698] Row locks acquired
0110 20:52:00.623198 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:00.623252 (+ 54us) write_op.cc:270] Start()
0110 20:52:01.254403 (+631151us) write_op.cc:276] Timestamp: P: 1736542320623192 usec, L: 0
0110 20:52:01.254409 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:01.254606 (+ 197us) log.cc:844] Serialized 354 byte log entry
0110 20:52:01.346580 (+ 91974us) op_driver.cc:464] REPLICATION: finished
0110 20:52:01.346674 (+ 94us) write_op.cc:301] APPLY: starting
0110 20:52:01.346698 (+ 24us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:01.346766 (+ 68us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:01.346794 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:01.346897 (+ 103us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:01.346941 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:01.346950 (+ 9us) write_op.cc:312] APPLY: finished
0110 20:52:01.347045 (+ 95us) log.cc:844] Serialized 22 byte log entry
0110 20:52:01.347121 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:01.347140 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:01.347147 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"num_ops":1,"prepare.queue_time_us":460686,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":631863,"raft.queue_time_us":39,"raft.run_cpu_time_us":20,"raft.run_wall_time_us":20,"replication_time_us":92133,"spinlock_wait_cycles":48256}]]}
I20250110 20:52:01.347858 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:01.348381 2554 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:01.981254 2036 tablet_service.cc:2970] error setting up scanner 5794872c6aff42138ee689e356786536 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.633s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
I20250110 20:52:02.075361 2569 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:02.075567 2569 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:02.076320 2569 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:02.076879 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:02.076898 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 28 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:02.077497 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:02.926293 2569 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:02.929234 2581 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=26, attempt_no=0}) took 1955 ms. Trace:
I20250110 20:52:02.929458 2581 rpcz_store.cc:276] 0110 20:52:00.973838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:00.973898 (+ 60us) service_pool.cc:225] Handling call
0110 20:52:02.929176 (+1955278us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:01.254787 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:01.254934 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:01.254941 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:01.254943 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:01.255065 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:01.255077 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:01.255081 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:01.255137 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:01.255140 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:01.255202 (+ 62us) write_op.cc:270] Start()
0110 20:52:02.020363 (+765161us) write_op.cc:276] Timestamp: P: 1736542321255135 usec, L: 0
0110 20:52:02.020369 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:02.020610 (+ 241us) log.cc:844] Serialized 354 byte log entry
0110 20:52:02.928143 (+907533us) op_driver.cc:464] REPLICATION: finished
0110 20:52:02.928555 (+ 412us) write_op.cc:301] APPLY: starting
0110 20:52:02.928572 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:02.928610 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:02.928655 (+ 45us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:02.928748 (+ 93us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:02.928782 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:02.928789 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:02.928948 (+ 159us) log.cc:844] Serialized 22 byte log entry
0110 20:52:02.929028 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:02.929047 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:02.929054 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"num_ops":1,"prepare.queue_time_us":280388,"prepare.run_cpu_time_us":827,"prepare.run_wall_time_us":765902,"replication_time_us":907741,"thread_start_us":259,"threads_started":1}]]}
I20250110 20:52:02.991988 2575 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:02.992198 2575 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:02.993306 2575 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:02.993693 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:02.993927 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 29 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:02.994531 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:03.113891 2575 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:03.115569 2581 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=27, attempt_no=0}) took 1766 ms. Trace:
I20250110 20:52:03.115711 2581 rpcz_store.cc:276] 0110 20:52:01.349297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:01.349360 (+ 63us) service_pool.cc:225] Handling call
0110 20:52:03.115501 (+1766141us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:02.020731 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:02.020859 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:02.020864 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:02.020866 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:02.020960 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:02.020972 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:02.020977 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:02.021034 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:52:02.021036 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:02.021085 (+ 49us) write_op.cc:270] Start()
0110 20:52:02.806239 (+785154us) write_op.cc:276] Timestamp: P: 1736542322021046 usec, L: 0
0110 20:52:02.806246 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:02.806504 (+ 258us) log.cc:844] Serialized 354 byte log entry
0110 20:52:03.114921 (+308417us) op_driver.cc:464] REPLICATION: finished
0110 20:52:03.115046 (+ 125us) write_op.cc:301] APPLY: starting
0110 20:52:03.115064 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:03.115100 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:03.115134 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:03.115200 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:03.115227 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:03.115232 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:03.115303 (+ 71us) log.cc:844] Serialized 22 byte log entry
0110 20:52:03.115380 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:03.115398 (+ 18us) write_op.cc:454] Released schema lock
0110 20:52:03.115405 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":52,"num_ops":1,"prepare.queue_time_us":670970,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":785853,"replication_time_us":308641}]]}
I20250110 20:52:03.447625 2569 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:03.447821 2569 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:03.448583 2569 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:03.449088 2077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:03.449173 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 30 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:03.449748 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:03.450541 2569 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:03.583781 2575 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:03.583989 2575 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:03.584920 2575 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:03.585741 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:03.586974 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:03.587539 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:03.587970 2575 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:03.593988 2431 scanner-internal.cc:458] Time spent opening tablet: real 20.795s user 0.007s sys 0.000s
W20250110 20:52:03.747072 2036 tablet_service.cc:2970] error setting up scanner a3bc9e1b1e704755912afefcbadbb8d1 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.161s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
W20250110 20:52:03.769217 2036 tablet_service.cc:2970] error setting up scanner a4db232464ba4f77bb7685e06b0bfea4 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.183s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:03.791520 2569 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:03.791710 2569 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:03.792466 2569 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:03.792831 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:03.793048 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:03.793820 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:03.794091 2569 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:03.808247 2036 tablet_service.cc:2970] error setting up scanner 264398dc95eb4c3585fc6d417d79fe10 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.222s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
W20250110 20:52:03.863838 2036 tablet_service.cc:2970] error setting up scanner 13fad32fa91444149146831c6bdfa980 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.277s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
W20250110 20:52:03.955085 2036 tablet_service.cc:2970] error setting up scanner eb412b4f3eee4e76ace45880dc7c083e with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.369s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:04.077975 2575 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:04.078203 2575 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:04.079027 2575 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:04.079723 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:04.079874 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 31 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:04.080715 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:04.081537 2575 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:04.261799 2036 tablet_service.cc:2970] error setting up scanner da4f8413ca814287921af8f14af2f5db with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.675s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:04.619503 2575 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:04.619696 2575 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:04.620400 2575 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:04.620843 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:04.620990 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:04.621634 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:04.622211 2593 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=29, attempt_no=0}) took 1504 ms. Trace:
I20250110 20:52:04.622419 2575 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:04.622331 2593 rpcz_store.cc:276] 0110 20:52:03.117515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:03.117587 (+ 72us) service_pool.cc:225] Handling call
0110 20:52:04.622159 (+1504572us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:03.555975 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:03.556103 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:03.556111 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:03.556113 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:03.556216 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:03.556227 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:03.556232 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:03.556286 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:52:03.556288 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:03.556337 (+ 49us) write_op.cc:270] Start()
0110 20:52:04.466479 (+910142us) write_op.cc:276] Timestamp: P: 1736542323556303 usec, L: 0
0110 20:52:04.466484 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:04.466697 (+ 213us) log.cc:844] Serialized 354 byte log entry
0110 20:52:04.620558 (+153861us) op_driver.cc:464] REPLICATION: finished
0110 20:52:04.621535 (+ 977us) write_op.cc:301] APPLY: starting
0110 20:52:04.621554 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:04.621605 (+ 51us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:04.621640 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:04.621739 (+ 99us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:04.621776 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:04.621784 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:04.621925 (+ 141us) log.cc:844] Serialized 22 byte log entry
0110 20:52:04.622005 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:04.622025 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:04.622032 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":894,"num_ops":1,"prepare.queue_time_us":437885,"prepare.run_cpu_time_us":795,"prepare.run_wall_time_us":910850,"raft.queue_time_us":25,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":110,"replication_time_us":154038,"spinlock_wait_cycles":248064,"thread_start_us":787,"threads_started":1}]]}
I20250110 20:52:04.766788 2575 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:04.767079 2575 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:04.767885 2575 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:04.768357 2593 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=30, attempt_no=0}) took 1181 ms. Trace:
I20250110 20:52:04.768353 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:04.768488 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 33 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:04.768592 2593 rpcz_store.cc:276] 0110 20:52:03.586828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:03.586869 (+ 41us) service_pool.cc:225] Handling call
0110 20:52:04.768286 (+1181417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:04.466872 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:04.467021 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:04.467028 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:04.467031 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:04.467133 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:04.467144 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:04.467149 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:04.467205 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:04.467208 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:04.467258 (+ 50us) write_op.cc:270] Start()
0110 20:52:04.650405 (+183147us) write_op.cc:276] Timestamp: P: 1736542324467242 usec, L: 0
0110 20:52:04.650410 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:04.650619 (+ 209us) log.cc:844] Serialized 354 byte log entry
0110 20:52:04.767622 (+117003us) op_driver.cc:464] REPLICATION: finished
0110 20:52:04.767755 (+ 133us) write_op.cc:301] APPLY: starting
0110 20:52:04.767775 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:04.767821 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:04.767864 (+ 43us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:04.767941 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:04.767977 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:04.767985 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:04.768077 (+ 92us) log.cc:844] Serialized 22 byte log entry
0110 20:52:04.768166 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:04.768185 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:04.768192 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":57,"num_ops":1,"prepare.queue_time_us":879513,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":183814,"replication_time_us":117178}]]}
I20250110 20:52:04.769052 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:04.769912 2575 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:05.313916 2590 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:05.314162 2590 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:05.315085 2590 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:05.315565 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:05.315632 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 32 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:05.316208 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:05.399050 2575 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:05.399266 2575 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:05.400044 2575 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:05.400460 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:05.401551 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 34 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:05.402010 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:05.402321 2575 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:05.713302 2036 tablet_service.cc:2970] error setting up scanner 6830151951ff4e22a12549b1662b6ecc with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736542324713063 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736542324651099 usec, L: 0 Physical time difference: 0.062s: new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:05.713750 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:46820 (request call id 34) took 1002 ms. Trace:
I20250110 20:52:05.713862 2036 rpcz_store.cc:276] 0110 20:52:04.711541 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:04.711633 (+ 92us) service_pool.cc:225] Handling call
0110 20:52:04.711886 (+ 253us) tablet_service.cc:2780] Created scanner 6830151951ff4e22a12549b1662b6ecc for tablet 66baaff8be8d49be99ab53b640904c2c, query id is ac16dcca4ec5415c94a209dba50fb470
0110 20:52:04.713038 (+ 1152us) tablet_service.cc:2920] Creating iterator
0110 20:52:04.713077 (+ 39us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:05.713241 (+1000164us) tablet_service.cc:2945] Iterator created
0110 20:52:05.713257 (+ 16us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736542324713063 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736542324651099 usec, L: 0 Physical time difference: 0.062s
0110 20:52:05.713723 (+ 466us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:52:06.337011 2036 tablet_service.cc:2970] error setting up scanner 6106ed036a114409a967882b290e3ea2 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.936s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
I20250110 20:52:06.337249 2590 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:06.337340 2207 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:34181 (request call id 3347) took 1021 ms. Trace:
I20250110 20:52:06.338076 2207 rpcz_store.cc:276] 0110 20:52:05.315701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:05.315785 (+ 84us) service_pool.cc:225] Handling call
0110 20:52:05.315990 (+ 205us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:52:05.316167 (+ 177us) raft_consensus.cc:1529] Early marking committed up to index 32
0110 20:52:05.316514 (+ 347us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:52:05.316683 (+ 169us) write_op.cc:270] Start()
0110 20:52:05.520865 (+204182us) write_op.cc:276] Timestamp: P: 1736542324467242 usec, L: 0
0110 20:52:05.521166 (+ 301us) write_op.cc:270] Start()
0110 20:52:06.335336 (+814170us) write_op.cc:276] Timestamp: P: 1736542324651099 usec, L: 0
0110 20:52:06.335992 (+ 656us) log.cc:844] Serialized 708 byte log entry
0110 20:52:06.336552 (+ 560us) raft_consensus.cc:1640] Marking committed up to 33
0110 20:52:06.337126 (+ 574us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:52:06.337165 (+ 39us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:52:06.337176 (+ 11us) raft_consensus.cc:1680] finished
0110 20:52:06.337178 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:52:06.337233 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:05.521059 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:05.521165 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:05.521171 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:05.521172 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:05.521264 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:05.521282 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:05.521287 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:05.521341 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:52:05.521344 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:06.336608 (+815264us) op_driver.cc:464] REPLICATION: finished
0110 20:52:06.337189 (+ 581us) write_op.cc:301] APPLY: starting
0110 20:52:06.337207 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:06.337330 (+ 123us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:06.337333 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:06.337404 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:06.337438 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:06.337496 (+ 58us) write_op.cc:312] APPLY: finished
0110 20:52:06.337572 (+ 76us) log.cc:844] Serialized 22 byte log entry
0110 20:52:06.337776 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:06.337793 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:06.337800 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 20:52:06.337891 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:06.337895 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:52:06.335836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:06.335940 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:06.335947 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:06.335948 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:06.336034 (+ 86us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:06.336047 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:06.336050 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:06.336098 (+ 48us) tablet.cc:698] Row locks acquired
0110 20:52:06.336099 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":466,"threads_started":1,"wal-append.queue_time_us":549,"child_traces":[["op",{"apply.queue_time_us":519,"apply.run_cpu_time_us":504,"apply.run_wall_time_us":701,"num_ops":1,"partition_lock_wait_us":9,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":384,"prepare.run_wall_time_us":383,"replication_time_us":1019896,"thread_start_us":447,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":345,"prepare.run_cpu_time_us":359,"prepare.run_wall_time_us":361,"thread_start_us":203,"threads_started":1}]]}
W20250110 20:52:06.868530 2036 tablet_service.cc:2970] error setting up scanner 155982957ba74c7283dd7979b57d73dd with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 1.467s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:06.970628 2590 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:06.970873 2590 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:06.971603 2590 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:06.972105 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:06.972220 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:06.972746 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:06.973037 2590 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:06.974776 2610 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=32, attempt_no=0}) took 2204 ms. Trace:
I20250110 20:52:06.974892 2610 rpcz_store.cc:276] 0110 20:52:04.770323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:04.770362 (+ 39us) service_pool.cc:225] Handling call
0110 20:52:06.974685 (+2204323us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:05.229573 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:05.229694 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:05.229701 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:05.229703 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:05.229807 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:05.229820 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:05.229825 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:05.229880 (+ 55us) tablet.cc:698] Row locks acquired
0110 20:52:05.229883 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:05.229928 (+ 45us) write_op.cc:270] Start()
0110 20:52:06.095071 (+865143us) write_op.cc:276] Timestamp: P: 1736542325229872 usec, L: 0
0110 20:52:06.095077 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:06.095261 (+ 184us) log.cc:844] Serialized 354 byte log entry
0110 20:52:06.973483 (+878222us) op_driver.cc:464] REPLICATION: finished
0110 20:52:06.974097 (+ 614us) write_op.cc:301] APPLY: starting
0110 20:52:06.974115 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:06.974158 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:06.974212 (+ 54us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:06.974327 (+ 115us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:06.974361 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:06.974368 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:06.974484 (+ 116us) log.cc:844] Serialized 22 byte log entry
0110 20:52:06.974566 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:06.974582 (+ 16us) write_op.cc:454] Released schema lock
0110 20:52:06.974589 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":545,"num_ops":1,"prepare.queue_time_us":458759,"prepare.run_cpu_time_us":685,"prepare.run_wall_time_us":865752,"replication_time_us":878364,"thread_start_us":440,"threads_started":1}]]}
I20250110 20:52:06.991103 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:06.991295 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:06.992096 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:06.992493 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:06.992568 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:06.993171 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:06.993440 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:07.080247 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:07.080399 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:07.080955 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:07.081501 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:07.081902 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 35 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:07.082369 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:07.083165 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:07.266516 2590 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:07.266767 2590 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:07.267513 2590 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:07.268097 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:07.268069 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:07.268337 2610 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=33, attempt_no=0}) took 1864 ms. Trace:
I20250110 20:52:07.268453 2610 rpcz_store.cc:276] 0110 20:52:05.403769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:05.403839 (+ 70us) service_pool.cc:225] Handling call
0110 20:52:07.268272 (+1864433us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:06.095367 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:06.095495 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:06.095502 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:06.095505 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:06.095598 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:06.095609 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:06.095614 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:06.095666 (+ 52us) tablet.cc:698] Row locks acquired
0110 20:52:06.095668 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:06.095711 (+ 43us) write_op.cc:270] Start()
0110 20:52:07.094821 (+999110us) write_op.cc:276] Timestamp: P: 1736542326095673 usec, L: 0
0110 20:52:07.094826 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:07.095037 (+ 211us) log.cc:844] Serialized 354 byte log entry
0110 20:52:07.267517 (+172480us) op_driver.cc:464] REPLICATION: finished
0110 20:52:07.267740 (+ 223us) write_op.cc:301] APPLY: starting
0110 20:52:07.267764 (+ 24us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:07.267808 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:07.267837 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:07.267945 (+ 108us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:07.267981 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:07.267988 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:07.268084 (+ 96us) log.cc:844] Serialized 22 byte log entry
0110 20:52:07.268151 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:07.268172 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:07.268179 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":123,"num_ops":1,"prepare.queue_time_us":691081,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":999742,"replication_time_us":172635}]]}
I20250110 20:52:07.268869 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:07.269433 2590 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:07.946604 2616 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:07.946841 2616 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:07.948124 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:07.948258 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 37 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:07.948765 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:07.948873 2616 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:07.949136 2616 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:07.955343 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:07.955556 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:07.956374 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:07.956843 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:07.957703 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 36 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:07.958253 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:08.552228 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:08.791452 2036 tablet_service.cc:2970] error setting up scanner ba0515461ad541d884692521050f240b with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.239s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:08.942696 2620 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:08.942946 2620 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:08.943783 2620 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:08.944468 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:08.944639 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:08.945449 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:08.946035 2620 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:08.948199 2630 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=35, attempt_no=0}) took 1678 ms. Trace:
I20250110 20:52:08.948318 2630 rpcz_store.cc:276] 0110 20:52:07.270124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:07.270195 (+ 71us) service_pool.cc:225] Handling call
0110 20:52:08.948141 (+1677946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:07.880032 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:07.880154 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:07.880160 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:07.880163 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:07.880266 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:07.880278 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:07.880283 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:07.880339 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:07.880341 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:07.880392 (+ 51us) write_op.cc:270] Start()
0110 20:52:08.012560 (+132168us) write_op.cc:276] Timestamp: P: 1736542327880325 usec, L: 0
0110 20:52:08.012566 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:08.012783 (+ 217us) log.cc:844] Serialized 354 byte log entry
0110 20:52:08.947088 (+934305us) op_driver.cc:464] REPLICATION: finished
0110 20:52:08.947535 (+ 447us) write_op.cc:301] APPLY: starting
0110 20:52:08.947574 (+ 39us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:08.947619 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:08.947645 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:08.947737 (+ 92us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:08.947772 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:08.947779 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:08.947879 (+ 100us) log.cc:844] Serialized 22 byte log entry
0110 20:52:08.947967 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:08.947999 (+ 32us) write_op.cc:454] Released schema lock
0110 20:52:08.948009 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":369,"num_ops":1,"prepare.queue_time_us":609341,"prepare.run_cpu_time_us":770,"prepare.run_wall_time_us":132827,"replication_time_us":934465,"thread_start_us":267,"threads_started":1}]]}
I20250110 20:52:09.272511 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:09.272724 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:09.273514 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:09.274008 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:09.274039 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 38 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:09.274637 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:09.693437 2620 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:09.693641 2620 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:09.694394 2620 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:09.694993 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:09.695559 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 39 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:09.696096 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:09.697166 2636 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=36, attempt_no=0}) took 1745 ms. Trace:
I20250110 20:52:09.697263 2636 rpcz_store.cc:276] 0110 20:52:07.951200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:07.951272 (+ 72us) service_pool.cc:225] Handling call
0110 20:52:09.697127 (+1745855us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:08.012902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:08.013039 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:08.013044 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:08.013046 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:08.013123 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:08.013132 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:08.013136 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:08.013180 (+ 44us) tablet.cc:698] Row locks acquired
0110 20:52:08.013181 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:08.013224 (+ 43us) write_op.cc:270] Start()
0110 20:52:08.799366 (+786142us) write_op.cc:276] Timestamp: P: 1736542328013188 usec, L: 0
0110 20:52:08.799372 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:08.799636 (+ 264us) log.cc:844] Serialized 354 byte log entry
0110 20:52:09.694871 (+895235us) op_driver.cc:464] REPLICATION: finished
0110 20:52:09.696685 (+ 1814us) write_op.cc:301] APPLY: starting
0110 20:52:09.696702 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:09.696732 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:09.696763 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:09.696830 (+ 67us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:09.696860 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:09.696867 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:09.696942 (+ 75us) log.cc:844] Serialized 22 byte log entry
0110 20:52:09.697036 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:09.697050 (+ 14us) write_op.cc:454] Released schema lock
0110 20:52:09.697056 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1753,"num_ops":1,"prepare.queue_time_us":61151,"prepare.run_cpu_time_us":753,"prepare.run_wall_time_us":786813,"replication_time_us":895458,"thread_start_us":1641,"threads_started":1}]]}
I20250110 20:52:10.020895 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:10.131835 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:10.132072 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:10.132854 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:10.133323 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:10.133551 2636 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=37, attempt_no=0}) took 1183 ms. Trace:
I20250110 20:52:10.133584 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 40 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:10.133690 2636 rpcz_store.cc:276] 0110 20:52:08.950177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:08.950242 (+ 65us) service_pool.cc:225] Handling call
0110 20:52:10.133481 (+1183239us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:08.950824 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:08.950949 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:08.950954 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:08.950957 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:08.951065 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:08.951074 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:08.951078 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:08.951122 (+ 44us) tablet.cc:698] Row locks acquired
0110 20:52:08.951123 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:08.951160 (+ 37us) write_op.cc:270] Start()
0110 20:52:09.568320 (+617160us) write_op.cc:276] Timestamp: P: 1736542328951096 usec, L: 0
0110 20:52:09.568325 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:09.568564 (+ 239us) log.cc:844] Serialized 354 byte log entry
0110 20:52:10.132821 (+564257us) op_driver.cc:464] REPLICATION: finished
0110 20:52:10.132969 (+ 148us) write_op.cc:301] APPLY: starting
0110 20:52:10.132989 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:10.133037 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:10.133071 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:10.133148 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:10.133183 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:10.133191 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:10.133283 (+ 92us) log.cc:844] Serialized 22 byte log entry
0110 20:52:10.133362 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:10.133382 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:10.133388 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"num_ops":1,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":755,"prepare.run_wall_time_us":617825,"replication_time_us":564459}]]}
I20250110 20:52:10.134333 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:10.134593 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:10.610512 2620 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:10.839622 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:10.839812 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:10.840529 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:10.841226 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:10.841946 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:10.842505 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:10.842916 2644 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=38, attempt_no=0}) took 1143 ms. Trace:
I20250110 20:52:10.843259 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:10.843075 2644 rpcz_store.cc:276] 0110 20:52:09.698966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:09.699032 (+ 66us) service_pool.cc:225] Handling call
0110 20:52:10.842858 (+1143826us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:09.699516 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:09.699615 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:09.699620 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:09.699621 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:09.699694 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:09.699701 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:09.699705 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:09.699750 (+ 45us) tablet.cc:698] Row locks acquired
0110 20:52:09.699752 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:09.699792 (+ 40us) write_op.cc:270] Start()
0110 20:52:10.206938 (+507146us) write_op.cc:276] Timestamp: P: 1736542329699737 usec, L: 0
0110 20:52:10.206944 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:10.207169 (+ 225us) log.cc:844] Serialized 354 byte log entry
0110 20:52:10.840963 (+633794us) op_driver.cc:464] REPLICATION: finished
0110 20:52:10.842013 (+ 1050us) write_op.cc:301] APPLY: starting
0110 20:52:10.842031 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:10.842067 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:10.842093 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:10.842177 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:10.842212 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:10.842220 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:10.842399 (+ 179us) log.cc:844] Serialized 22 byte log entry
0110 20:52:10.842689 (+ 290us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:10.842710 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:10.842718 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":982,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":658,"prepare.run_wall_time_us":507724,"replication_time_us":633981,"thread_start_us":1266,"threads_started":1}]]}
W20250110 20:52:10.948099 2036 tablet_service.cc:2970] error setting up scanner f8593f24d22d4338b217a66be7fbafe0 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.106s (maximum allowed 0.100s): new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "b8a8b18cc6c547ee99ba0169ad498ce5"
I20250110 20:52:11.589712 2620 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:11.589928 2620 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:11.590777 2620 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:11.591203 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:11.591212 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 41 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:11.591866 2140 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:11.885113 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:11.885306 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:11.886080 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:11.886466 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:11.887229 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 42 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:11.887869 2651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=39, attempt_no=0}) took 1752 ms. Trace:
I20250110 20:52:11.888038 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:11.888020 2651 rpcz_store.cc:276] 0110 20:52:10.135492 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:10.135573 (+ 81us) service_pool.cc:225] Handling call
0110 20:52:11.887813 (+1752240us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:10.207280 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:10.207402 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:10.207409 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:10.207412 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:10.207513 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:10.207525 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:10.207530 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:10.207584 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:52:10.207587 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:10.207636 (+ 49us) write_op.cc:270] Start()
0110 20:52:11.201785 (+994149us) write_op.cc:276] Timestamp: P: 1736542330207595 usec, L: 0
0110 20:52:11.201792 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:11.202000 (+ 208us) log.cc:844] Serialized 354 byte log entry
0110 20:52:11.886531 (+684531us) op_driver.cc:464] REPLICATION: finished
0110 20:52:11.887173 (+ 642us) write_op.cc:301] APPLY: starting
0110 20:52:11.887192 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:11.887241 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:11.887270 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:11.887383 (+ 113us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:11.887417 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:11.887423 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:11.887555 (+ 132us) log.cc:844] Serialized 22 byte log entry
0110 20:52:11.887668 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:11.887691 (+ 23us) write_op.cc:454] Released schema lock
0110 20:52:11.887698 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":564,"num_ops":1,"prepare.queue_time_us":71282,"prepare.run_cpu_time_us":729,"prepare.run_wall_time_us":994784,"replication_time_us":684689,"thread_start_us":1487,"threads_started":1}]]}
I20250110 20:52:11.888451 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:11.929926 2620 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:12.214587 2620 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:12.214805 2620 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:12.215588 2620 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:12.216056 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:12.216143 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:12.216326 2651 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=40, attempt_no=0}) took 1371 ms. Trace:
I20250110 20:52:12.216691 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:12.216455 2651 rpcz_store.cc:276] 0110 20:52:10.844841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:10.844899 (+ 58us) service_pool.cc:225] Handling call
0110 20:52:12.216236 (+1371337us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:11.202095 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:11.202232 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:11.202239 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:11.202242 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:11.202339 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:11.202349 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:11.202352 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:11.202397 (+ 45us) tablet.cc:698] Row locks acquired
0110 20:52:11.202399 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:11.202440 (+ 41us) write_op.cc:270] Start()
0110 20:52:11.910601 (+708161us) write_op.cc:276] Timestamp: P: 1736542331202401 usec, L: 0
0110 20:52:11.910606 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:11.910872 (+ 266us) log.cc:844] Serialized 354 byte log entry
0110 20:52:12.215596 (+304724us) op_driver.cc:464] REPLICATION: finished
0110 20:52:12.215717 (+ 121us) write_op.cc:301] APPLY: starting
0110 20:52:12.215736 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:12.215782 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:12.215819 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:12.215890 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:12.215924 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:12.215931 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:12.216030 (+ 99us) log.cc:844] Serialized 22 byte log entry
0110 20:52:12.216107 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:12.216130 (+ 23us) write_op.cc:454] Released schema lock
0110 20:52:12.216137 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":63,"num_ops":1,"prepare.queue_time_us":356783,"prepare.run_cpu_time_us":795,"prepare.run_wall_time_us":708849,"replication_time_us":304955}]]}
I20250110 20:52:12.265108 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:12.265359 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:12.266119 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:12.266584 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:12.266686 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 43 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:12.267325 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:12.314083 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:13.149128 2620 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:13.288827 2603 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:13.289079 2603 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:13.290060 2603 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:13.290550 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:13.290661 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:13.291282 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:13.291622 2658 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=42, attempt_no=0}) took 1073 ms. Trace:
I20250110 20:52:13.291755 2658 rpcz_store.cc:276] 0110 20:52:12.218283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:12.218355 (+ 72us) service_pool.cc:225] Handling call
0110 20:52:13.291559 (+1073204us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:12.218838 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:12.218931 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:12.218935 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:52:12.218937 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:12.219009 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:12.219018 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:12.219021 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:12.219081 (+ 60us) tablet.cc:698] Row locks acquired
0110 20:52:12.219083 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:12.219118 (+ 35us) write_op.cc:270] Start()
0110 20:52:12.416378 (+197260us) write_op.cc:276] Timestamp: P: 1736542332219089 usec, L: 0
0110 20:52:12.416384 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:12.416621 (+ 237us) log.cc:844] Serialized 354 byte log entry
0110 20:52:13.290195 (+873574us) op_driver.cc:464] REPLICATION: finished
0110 20:52:13.290877 (+ 682us) write_op.cc:301] APPLY: starting
0110 20:52:13.290897 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:13.290941 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:13.291112 (+ 171us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:13.291205 (+ 93us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:13.291241 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:13.291248 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:13.291344 (+ 96us) log.cc:844] Serialized 22 byte log entry
0110 20:52:13.291432 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:13.291454 (+ 22us) write_op.cc:454] Released schema lock
0110 20:52:13.291462 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":603,"num_ops":1,"prepare.queue_time_us":69,"prepare.run_cpu_time_us":698,"prepare.run_wall_time_us":197863,"replication_time_us":873753,"thread_start_us":464,"threads_started":1}]]}
W20250110 20:52:13.378351 2036 tablet_service.cc:2970] error setting up scanner 6e50f95024d44305934ed96534677864 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736542332378085 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736542332219089 usec, L: 0 Physical time difference: 0.159s: new_scan_request { tablet_id: "66baaff8be8d49be99ab53b640904c2c" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT propagated_timestamp: 7112877357436829696 cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\212\223\206\274\006\"3\n\005slave\022*\n 81ad58d7b1a44f989cb245bc4020edeb\020\001\030\001 \001(\001" signature: "<redacted>""\npd\235\253eLj\177\000\365Bi\305P\334\307UI\327\033\235\023\363\372\360\021z}\217)\236=c\267\362Z\250U\224 ,\003y\340\361Jp\371\217=\007}:\205\303\371\242\\\264c\234\343\256" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ac16dcca4ec5415c94a209dba50fb470"
I20250110 20:52:13.378944 2036 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:46820 (request call id 39) took 1001 ms. Trace:
I20250110 20:52:13.379037 2036 rpcz_store.cc:276] 0110 20:52:12.377086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:12.377143 (+ 57us) service_pool.cc:225] Handling call
0110 20:52:12.377357 (+ 214us) tablet_service.cc:2780] Created scanner 6e50f95024d44305934ed96534677864 for tablet 66baaff8be8d49be99ab53b640904c2c, query id is ac16dcca4ec5415c94a209dba50fb470
0110 20:52:12.378074 (+ 717us) tablet_service.cc:2920] Creating iterator
0110 20:52:12.378105 (+ 31us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:13.378298 (+1000193us) tablet_service.cc:2945] Iterator created
0110 20:52:13.378315 (+ 17us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736542332378085 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736542332219089 usec, L: 0 Physical time difference: 0.159s
0110 20:52:13.378879 (+ 564us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250110 20:52:13.817757 2620 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:13.817974 2620 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:13.819073 2620 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:13.819474 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:13.819706 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:13.820327 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:13.821249 2620 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:13.822238 2662 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:56756 (ReqId={client: b45aecd06c0d44f282456edd1c788d8e, seq_no=43, attempt_no=0}) took 1505 ms. Trace:
I20250110 20:52:13.822391 2662 rpcz_store.cc:276] 0110 20:52:12.317171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:12.317230 (+ 59us) service_pool.cc:225] Handling call
0110 20:52:13.822175 (+1504945us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:12.416753 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 66baaff8be8d49be99ab53b640904c2c
0110 20:52:12.416868 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:12.416875 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:12.416878 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:12.416985 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:12.416997 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:12.417002 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:12.417060 (+ 58us) tablet.cc:698] Row locks acquired
0110 20:52:12.417062 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:12.417113 (+ 51us) write_op.cc:270] Start()
0110 20:52:13.146271 (+729158us) write_op.cc:276] Timestamp: P: 1736542332417059 usec, L: 0
0110 20:52:13.146276 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:13.146515 (+ 239us) log.cc:844] Serialized 354 byte log entry
0110 20:52:13.819347 (+672832us) op_driver.cc:464] REPLICATION: finished
0110 20:52:13.821684 (+ 2337us) write_op.cc:301] APPLY: starting
0110 20:52:13.821699 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:13.821729 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:13.821756 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:13.821817 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:13.821845 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:13.821860 (+ 15us) write_op.cc:312] APPLY: finished
0110 20:52:13.821972 (+ 112us) log.cc:844] Serialized 22 byte log entry
0110 20:52:13.822063 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:13.822083 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:13.822090 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2274,"num_ops":1,"prepare.queue_time_us":98971,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":729834,"replication_time_us":673033,"thread_start_us":2192,"threads_started":1}]]}
I20250110 20:52:14.038831 2603 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:14.283501 2430 scanner-internal.cc:458] Time spent opening tablet: real 31.485s user 0.006s sys 0.003s
I20250110 20:52:15.754546 2668 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:15.754765 2668 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:15.754838 2669 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:15.755048 2669 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:15.755519 2668 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:15.756250 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:15.756383 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:15.756481 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:15.756870 2669 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:15.757400 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:15.757930 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:15.757987 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:15.758229 2669 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:15.758680 2668 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:15.807677 2668 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:15.807864 2668 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:15.808554 2668 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:15.809181 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:15.809212 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:15.809916 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:15.810298 2668 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:16.009220 2669 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:16.009428 2669 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:16.010146 2669 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:16.010512 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:16.010707 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:16.011314 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:16.011727 2669 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:52:16.969157 2431 scanner-internal.cc:458] Time spent opening tablet: real 13.224s user 0.006s sys 0.000s
I20250110 20:52:17.039671 2677 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:17.039868 2677 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:17.040475 2677 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:17.040777 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:17.041052 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:17.041616 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:17.042593 2677 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:17.098389 2187 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:52:17.105628 2057 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:52:17.106863 2317 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:52:17.107079 2677 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:17.107234 2677 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:17.107741 2677 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:17.108434 2337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:17.109342 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:17.109803 2011 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:17.110100 2677 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:17.123420 2697 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:17.123613 2697 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:17.124605 2075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" is_pre_election: true
I20250110 20:52:17.127822 2697 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:17.127915 2336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "555db90ad09d47708f85e4658c394115" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:17.130012 2143 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 555db90ad09d47708f85e4658c394115; no voters: 95d4d0dd0b954191b938a467d41914e7, 9ea09a0a2f2549a4a06723c0478f03f1
I20250110 20:52:17.130555 2697 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
7f090285d86b4c2593b052bc593f8678 | 127.0.104.62:35383 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:46315 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
555db90ad09d47708f85e4658c394115 | 127.0.104.2:40905 | HEALTHY | <none> | 0 | 0
95d4d0dd0b954191b938a467d41914e7 | 127.0.104.3:44233 | HEALTHY | <none> | 1 | 0
9ea09a0a2f2549a4a06723c0478f03f1 | 127.0.104.1:35577 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:35577
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:40905
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:44233
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:35577
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:40905
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:44233
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:46315 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 20:52:17.161898 416 log_verifier.cc:126] Checking tablet 66baaff8be8d49be99ab53b640904c2c
I20250110 20:52:17.170981 2677 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:17.171168 2677 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:17.171818 2677 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:17.172291 2207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:17.173067 2336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:17.173480 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
I20250110 20:52:17.173920 2677 raft_consensus.cc:2743] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:17.216399 416 log_verifier.cc:177] Verified matching terms for 46 ops in tablet 66baaff8be8d49be99ab53b640904c2c
I20250110 20:52:17.248224 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1995
I20250110 20:52:17.258021 2117 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:52:17.459700 2677 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
I20250110 20:52:17.459893 2677 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:17.460443 2677 leader_election.cc:290] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905), 95d4d0dd0b954191b938a467d41914e7 (127.0.104.3:44233)
I20250110 20:52:17.461498 2206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "555db90ad09d47708f85e4658c394115" is_pre_election: true
I20250110 20:52:17.465833 2336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66baaff8be8d49be99ab53b640904c2c" candidate_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "95d4d0dd0b954191b938a467d41914e7" is_pre_election: true
I20250110 20:52:17.466428 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1995
I20250110 20:52:17.466454 2013 leader_election.cc:304] T 66baaff8be8d49be99ab53b640904c2c P 9ea09a0a2f2549a4a06723c0478f03f1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ea09a0a2f2549a4a06723c0478f03f1; no voters: 555db90ad09d47708f85e4658c394115, 95d4d0dd0b954191b938a467d41914e7
W20250110 20:52:17.502436 2270 connection.cc:537] client connection to 127.0.104.1:35577 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:52:17.502653 2270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 2 similar messages]
I20250110 20:52:17.502868 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 2125
W20250110 20:52:17.504765 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:52:17.509738 2247 generic_service.cc:196] Checking for leaks (request via RPC)
W20250110 20:52:17.516436 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250110 20:52:17.527758 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250110 20:52:17.539012 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20250110 20:52:17.550114 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20250110 20:52:17.561407 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20250110 20:52:17.572702 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20250110 20:52:17.584071 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20250110 20:52:17.595184 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20250110 20:52:17.606482 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20250110 20:52:17.617478 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20250110 20:52:17.628930 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20250110 20:52:17.640388 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20250110 20:52:17.652411 2697 raft_consensus.cc:491] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 95d4d0dd0b954191b938a467d41914e7)
W20250110 20:52:17.652617 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20250110 20:52:17.652640 2697 raft_consensus.cc:513] T 66baaff8be8d49be99ab53b640904c2c P 555db90ad09d47708f85e4658c394115 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ea09a0a2f2549a4a06723c0478f03f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35577 } } peers { permanent_uuid: "555db90ad09d47708f85e4658c394115" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40905 } } peers { permanent_uuid: "95d4d0dd0b954191b938a467d41914e7" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44233 } }
I20250110 20:52:17.652916 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 2125
W20250110 20:52:17.664032 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20250110 20:52:17.675444 2270 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 9ea09a0a2f2549a4a06723c0478f03f1 (127.0.104.1:35577): Couldn't send request to peer 9ea09a0a2f2549a4a06723c0478f03f1. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35577: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20250110 20:52:17.679000 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 2255
W20250110 20:52:17.679291 2271 consensus_peers.cc:487] T 66baaff8be8d49be99ab53b640904c2c P 95d4d0dd0b954191b938a467d41914e7 -> Peer 555db90ad09d47708f85e4658c394115 (127.0.104.2:40905): Couldn't send request to peer 555db90ad09d47708f85e4658c394115. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.2:40905: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:52:17.686183 2377 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:52:17.870200 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 2255
I20250110 20:52:17.893468 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 1905
I20250110 20:52:17.894493 1966 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:52:18.002460 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 1905
2025-01-10T20:52:18Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (37376 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20250110 20:52:18.041137 416 test_util.cc:274] Using random seed: 1160922371
I20250110 20:52:18.043146 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:52:18.043236 416 ts_itest-base.cc:113] --------------
I20250110 20:52:18.043277 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:52:18.043334 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:52:18.043375 416 ts_itest-base.cc:116] --------------
2025-01-10T20:52:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:52:18Z Disabled control of system clock
I20250110 20:52:18.068905 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34885
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34885 with env {}
W20250110 20:52:18.397102 2721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:18.397413 2721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:18.397539 2721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:18.405277 2721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:52:18.405411 2721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:18.405474 2721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:52:18.405546 2721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:52:18.414954 2721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34885
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:18.416577 2721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:18.418334 2721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:18.428388 2726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:18.428341 2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:18.428411 2727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:18.429102 2721 server_base.cc:1034] running on GCE node
I20250110 20:52:18.429935 2721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:18.431453 2721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:18.432698 2721 hybrid_clock.cc:648] HybridClock initialized: now 1736542338432644 us; error 37 us; skew 500 ppm
I20250110 20:52:18.433029 2721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:18.435387 2721 webserver.cc:458] Webserver started at http://127.0.104.62:36355/ using document root <none> and password file <none>
I20250110 20:52:18.436120 2721 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:18.436236 2721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:18.436525 2721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:18.439319 2721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0ec47a07211e43cda060a0addfeacf36"
format_stamp: "Formatted at 2025-01-10 20:52:18 on dist-test-slave-1h8z"
I20250110 20:52:18.440104 2721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0ec47a07211e43cda060a0addfeacf36"
format_stamp: "Formatted at 2025-01-10 20:52:18 on dist-test-slave-1h8z"
I20250110 20:52:18.445451 2721 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:52:18.449059 2735 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:18.450547 2721 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:52:18.450811 2721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0ec47a07211e43cda060a0addfeacf36"
format_stamp: "Formatted at 2025-01-10 20:52:18 on dist-test-slave-1h8z"
I20250110 20:52:18.451014 2721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:18.483850 2721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:18.484717 2721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:18.484992 2721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:18.501754 2721 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34885
I20250110 20:52:18.501783 2786 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34885 every 8 connection(s)
I20250110 20:52:18.503149 2721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:52:18.506923 2787 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:18.507236 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 2721
I20250110 20:52:18.507413 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:52:18.518352 2787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36: Bootstrap starting.
I20250110 20:52:18.521833 2787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:18.522822 2787 log.cc:826] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:18.525365 2787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36: No bootstrap required, opened a new log
I20250110 20:52:18.531270 2787 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec47a07211e43cda060a0addfeacf36" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34885 } }
I20250110 20:52:18.531627 2787 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:18.531723 2787 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec47a07211e43cda060a0addfeacf36, State: Initialized, Role: FOLLOWER
I20250110 20:52:18.532286 2787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec47a07211e43cda060a0addfeacf36" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34885 } }
I20250110 20:52:18.532523 2787 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:52:18.532656 2787 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:52:18.532841 2787 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:18.534869 2787 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec47a07211e43cda060a0addfeacf36" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34885 } }
I20250110 20:52:18.535337 2787 leader_election.cc:304] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec47a07211e43cda060a0addfeacf36; no voters:
I20250110 20:52:18.535881 2787 leader_election.cc:290] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:52:18.536031 2792 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:52:18.536573 2792 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec47a07211e43cda060a0addfeacf36, State: Running, Role: LEADER
I20250110 20:52:18.537086 2792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec47a07211e43cda060a0addfeacf36" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34885 } }
I20250110 20:52:18.537796 2787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:52:18.539691 2793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec47a07211e43cda060a0addfeacf36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec47a07211e43cda060a0addfeacf36" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34885 } } }
I20250110 20:52:18.539999 2793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [sys.catalog]: This master's current role is: LEADER
I20250110 20:52:18.540306 2794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec47a07211e43cda060a0addfeacf36. Latest consensus state: current_term: 1 leader_uuid: "0ec47a07211e43cda060a0addfeacf36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec47a07211e43cda060a0addfeacf36" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34885 } } }
I20250110 20:52:18.540539 2794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36 [sys.catalog]: This master's current role is: LEADER
I20250110 20:52:18.543991 2798 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:52:18.549384 2798 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:52:18.558431 2798 catalog_manager.cc:1348] Generated new cluster ID: 603a75441f0143cf9a7c012c7d07fefe
I20250110 20:52:18.558578 2798 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:52:18.578142 2798 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:52:18.579898 2798 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:52:18.590158 2798 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0ec47a07211e43cda060a0addfeacf36: Generated new TSK 0
I20250110 20:52:18.591181 2798 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:52:18.601514 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34885
--builtin_ntp_servers=127.0.104.20:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20250110 20:52:18.917402 2811 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20250110 20:52:18.917714 2811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:18.917801 2811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:18.917918 2811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:18.927079 2811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:18.927362 2811 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:52:18.937999 2811 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:34885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:18.939834 2811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:18.941466 2811 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:18.952996 2819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:18.953052 2817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:18.953169 2816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:18.953823 2811 server_base.cc:1034] running on GCE node
I20250110 20:52:18.954401 2811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:18.955384 2811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:18.956614 2811 hybrid_clock.cc:648] HybridClock initialized: now 1736542338956549 us; error 49 us; skew 500 ppm
I20250110 20:52:18.957022 2811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:18.959450 2811 webserver.cc:458] Webserver started at http://127.0.104.1:36513/ using document root <none> and password file <none>
I20250110 20:52:18.960206 2811 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:18.960297 2811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:18.960595 2811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:18.963267 2811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2ec9211af4dd4e1f9eb696659f945644"
format_stamp: "Formatted at 2025-01-10 20:52:18 on dist-test-slave-1h8z"
I20250110 20:52:18.964023 2811 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2ec9211af4dd4e1f9eb696659f945644"
format_stamp: "Formatted at 2025-01-10 20:52:18 on dist-test-slave-1h8z"
I20250110 20:52:18.969156 2811 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:52:18.972406 2825 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:18.973721 2811 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250110 20:52:18.973873 2811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2ec9211af4dd4e1f9eb696659f945644"
format_stamp: "Formatted at 2025-01-10 20:52:18 on dist-test-slave-1h8z"
I20250110 20:52:18.974058 2811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:18.996004 2811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:18.996840 2811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:18.997092 2811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:18.998010 2811 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:52:18.999563 2811 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:52:18.999652 2811 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:18.999755 2811 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:52:18.999820 2811 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.029049 2811 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39435
I20250110 20:52:19.029153 2937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39435 every 8 connection(s)
I20250110 20:52:19.030432 2811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:52:19.039535 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 2811
I20250110 20:52:19.039762 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:52:19.043367 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34885
--builtin_ntp_servers=127.0.104.20:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20250110 20:52:19.043788 2938 heartbeater.cc:346] Connected to a master server at 127.0.104.62:34885
I20250110 20:52:19.044095 2938 heartbeater.cc:463] Registering TS with master...
I20250110 20:52:19.044946 2938 heartbeater.cc:510] Master 127.0.104.62:34885 requested a full tablet report, sending...
I20250110 20:52:19.047266 2752 ts_manager.cc:194] Registered new tserver with Master: 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:19.048928 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:46513
W20250110 20:52:19.370690 2942 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20250110 20:52:19.371101 2942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:19.371189 2942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:19.371306 2942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:19.379208 2942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:19.379439 2942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:52:19.390055 2942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:34885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:19.391744 2942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:19.393507 2942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:19.405126 2950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:19.405166 2947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:19.405194 2948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:19.406219 2942 server_base.cc:1034] running on GCE node
I20250110 20:52:19.406852 2942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:19.407829 2942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:19.409034 2942 hybrid_clock.cc:648] HybridClock initialized: now 1736542339408972 us; error 61 us; skew 500 ppm
I20250110 20:52:19.409360 2942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:19.411762 2942 webserver.cc:458] Webserver started at http://127.0.104.2:46781/ using document root <none> and password file <none>
I20250110 20:52:19.412525 2942 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:19.412639 2942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:19.412961 2942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:19.415746 2942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "730f5dd0bdf9484f805e503c0e4754f7"
format_stamp: "Formatted at 2025-01-10 20:52:19 on dist-test-slave-1h8z"
I20250110 20:52:19.416517 2942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "730f5dd0bdf9484f805e503c0e4754f7"
format_stamp: "Formatted at 2025-01-10 20:52:19 on dist-test-slave-1h8z"
I20250110 20:52:19.421929 2942 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250110 20:52:19.425402 2956 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.427042 2942 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:52:19.427198 2942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "730f5dd0bdf9484f805e503c0e4754f7"
format_stamp: "Formatted at 2025-01-10 20:52:19 on dist-test-slave-1h8z"
I20250110 20:52:19.427379 2942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:19.458148 2942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:19.459072 2942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:19.459316 2942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:19.460276 2942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:52:19.461805 2942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:52:19.461901 2942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.461988 2942 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:52:19.462040 2942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.491891 2942 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44259
I20250110 20:52:19.491945 3068 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44259 every 8 connection(s)
I20250110 20:52:19.493270 2942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:52:19.503365 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 2942
I20250110 20:52:19.503600 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:52:19.506641 3069 heartbeater.cc:346] Connected to a master server at 127.0.104.62:34885
I20250110 20:52:19.507006 3069 heartbeater.cc:463] Registering TS with master...
I20250110 20:52:19.507025 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34885
--builtin_ntp_servers=127.0.104.20:44999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20250110 20:52:19.508034 3069 heartbeater.cc:510] Master 127.0.104.62:34885 requested a full tablet report, sending...
I20250110 20:52:19.509735 2752 ts_manager.cc:194] Registered new tserver with Master: 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:19.510779 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:34369
W20250110 20:52:19.844116 3072 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20250110 20:52:19.844444 3072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:19.844516 3072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:19.844601 3072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:19.852474 3072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:19.852698 3072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:52:19.862376 3072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:34885
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:19.864061 3072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:19.865864 3072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:19.877912 3080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:19.877871 3078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:19.877939 3077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:19.878819 3072 server_base.cc:1034] running on GCE node
I20250110 20:52:19.879490 3072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:19.880497 3072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:19.881731 3072 hybrid_clock.cc:648] HybridClock initialized: now 1736542339881658 us; error 53 us; skew 500 ppm
I20250110 20:52:19.882069 3072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:19.884457 3072 webserver.cc:458] Webserver started at http://127.0.104.3:36739/ using document root <none> and password file <none>
I20250110 20:52:19.885253 3072 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:19.885370 3072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:19.885695 3072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:19.888589 3072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "372d84f3d6134989b052a99839ab6a65"
format_stamp: "Formatted at 2025-01-10 20:52:19 on dist-test-slave-1h8z"
I20250110 20:52:19.889405 3072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "372d84f3d6134989b052a99839ab6a65"
format_stamp: "Formatted at 2025-01-10 20:52:19 on dist-test-slave-1h8z"
I20250110 20:52:19.894973 3072 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250110 20:52:19.898552 3086 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.900048 3072 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:52:19.900216 3072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "372d84f3d6134989b052a99839ab6a65"
format_stamp: "Formatted at 2025-01-10 20:52:19 on dist-test-slave-1h8z"
I20250110 20:52:19.900421 3072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:19.931761 3072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:19.932760 3072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:19.933048 3072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:19.934072 3072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:52:19.935705 3072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:52:19.935814 3072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.935935 3072 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:52:19.936002 3072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:19.965533 3072 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44505
I20250110 20:52:19.965595 3198 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44505 every 8 connection(s)
I20250110 20:52:19.966958 3072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:52:19.969338 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 3072
I20250110 20:52:19.969516 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:52:19.979899 3199 heartbeater.cc:346] Connected to a master server at 127.0.104.62:34885
I20250110 20:52:19.980217 3199 heartbeater.cc:463] Registering TS with master...
I20250110 20:52:19.981065 3199 heartbeater.cc:510] Master 127.0.104.62:34885 requested a full tablet report, sending...
I20250110 20:52:19.982594 2752 ts_manager.cc:194] Registered new tserver with Master: 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505)
I20250110 20:52:19.983431 2752 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:47561
I20250110 20:52:19.985152 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:52:19.988509 416 test_util.cc:274] Using random seed: 1162869767
I20250110 20:52:20.006918 2752 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45852:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:52:20.008502 2752 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:52:20.038867 3134 tablet_service.cc:1467] Processing CreateTablet for tablet 257e9d67bed34c4990d8d399fde6e9e3 (DEFAULT_TABLE table=test-workload [id=20f4745234e8434fbc5ec5e0c2d40f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:52:20.040632 3134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 257e9d67bed34c4990d8d399fde6e9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:20.044175 2873 tablet_service.cc:1467] Processing CreateTablet for tablet 257e9d67bed34c4990d8d399fde6e9e3 (DEFAULT_TABLE table=test-workload [id=20f4745234e8434fbc5ec5e0c2d40f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:52:20.045688 2873 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 257e9d67bed34c4990d8d399fde6e9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:20.048638 3004 tablet_service.cc:1467] Processing CreateTablet for tablet 257e9d67bed34c4990d8d399fde6e9e3 (DEFAULT_TABLE table=test-workload [id=20f4745234e8434fbc5ec5e0c2d40f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:52:20.050235 3004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 257e9d67bed34c4990d8d399fde6e9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:20.051848 2938 heartbeater.cc:502] Master 127.0.104.62:34885 was elected leader, sending a full tablet report...
I20250110 20:52:20.054669 3217 tablet_bootstrap.cc:492] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65: Bootstrap starting.
I20250110 20:52:20.058960 3217 tablet_bootstrap.cc:654] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:20.059341 3218 tablet_bootstrap.cc:492] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644: Bootstrap starting.
I20250110 20:52:20.060340 3217 log.cc:826] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:20.062956 3218 tablet_bootstrap.cc:654] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:20.063131 3220 tablet_bootstrap.cc:492] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7: Bootstrap starting.
I20250110 20:52:20.063458 3217 tablet_bootstrap.cc:492] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65: No bootstrap required, opened a new log
I20250110 20:52:20.063818 3217 ts_tablet_manager.cc:1397] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20250110 20:52:20.064378 3218 log.cc:826] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:20.066542 3220 tablet_bootstrap.cc:654] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:20.067611 3220 log.cc:826] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:20.070259 3218 tablet_bootstrap.cc:492] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644: No bootstrap required, opened a new log
I20250110 20:52:20.070390 3220 tablet_bootstrap.cc:492] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7: No bootstrap required, opened a new log
I20250110 20:52:20.070549 3218 ts_tablet_manager.cc:1397] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20250110 20:52:20.070667 3220 ts_tablet_manager.cc:1397] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250110 20:52:20.072779 3217 raft_consensus.cc:357] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.073308 3217 raft_consensus.cc:383] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:20.073441 3217 raft_consensus.cc:738] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372d84f3d6134989b052a99839ab6a65, State: Initialized, Role: FOLLOWER
I20250110 20:52:20.074225 3217 consensus_queue.cc:260] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.076084 3217 ts_tablet_manager.cc:1428] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20250110 20:52:20.077080 3224 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:20.077190 3199 heartbeater.cc:502] Master 127.0.104.62:34885 was elected leader, sending a full tablet report...
I20250110 20:52:20.077351 3224 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.079391 3218 raft_consensus.cc:357] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.079960 3218 raft_consensus.cc:383] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:20.080114 3218 raft_consensus.cc:738] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ec9211af4dd4e1f9eb696659f945644, State: Initialized, Role: FOLLOWER
I20250110 20:52:20.080852 3218 consensus_queue.cc:260] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.081828 3226 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:20.082105 3226 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.085090 3226 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
W20250110 20:52:20.087849 3088 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.1:39435 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
I20250110 20:52:20.090368 3218 ts_tablet_manager.cc:1428] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644: Time spent starting tablet: real 0.020s user 0.011s sys 0.001s
I20250110 20:52:20.095642 3224 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:20.102293 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:20.102833 2893 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372d84f3d6134989b052a99839ab6a65 in term 0.
I20250110 20:52:20.102706 3220 raft_consensus.cc:357] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.103451 3220 raft_consensus.cc:383] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:20.103623 3220 raft_consensus.cc:738] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 730f5dd0bdf9484f805e503c0e4754f7, State: Initialized, Role: FOLLOWER
I20250110 20:52:20.103621 3088 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65; no voters:
I20250110 20:52:20.104176 3224 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:52:20.104393 3224 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250110 20:52:20.104457 2828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.2:44259 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20250110 20:52:20.106696 3224 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:20.110325 3224 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.114485 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:20.116677 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644"
I20250110 20:52:20.116942 2893 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:20.120803 2893 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372d84f3d6134989b052a99839ab6a65 in term 1.
I20250110 20:52:20.121541 3088 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65; no voters:
I20250110 20:52:20.121991 3223 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 20:52:20.122478 3089 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Timed out: RequestConsensusVote RPC to 127.0.104.2:44259 timed out after 0.008s (SENT)
I20250110 20:52:20.122620 3223 raft_consensus.cc:695] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 1 LEADER]: Becoming Leader. State: Replica: 372d84f3d6134989b052a99839ab6a65, State: Running, Role: LEADER
I20250110 20:52:20.123250 3223 consensus_queue.cc:237] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.126937 3224 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 1 election: Requested vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:20.128037 3023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7"
I20250110 20:52:20.130476 2752 catalog_manager.cc:5526] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 372d84f3d6134989b052a99839ab6a65 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "372d84f3d6134989b052a99839ab6a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } health_report { overall_health: UNKNOWN } } }
I20250110 20:52:20.131644 3220 consensus_queue.cc:260] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.132277 3024 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372d84f3d6134989b052a99839ab6a65 in term 0.
W20250110 20:52:20.133298 3024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.3:51333 (request call id 1) took 19 ms (client timeout 8 ms). Trace:
W20250110 20:52:20.133424 3024 rpcz_store.cc:269] 0110 20:52:20.113820 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:20.113947 (+ 127us) service_pool.cc:225] Handling call
0110 20:52:20.133179 (+ 19232us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x616000071554. stack: 00007fe12398bfd6 00007fe12398bd90 00007fe12296b48d 000000000096d544 000000000096d50d 000000000096ceda 00007fe12f21fb85 00007fe132f1cce4 00007fe12d5ef9b6 00007fe12d5ef6e5 00007fe12a6f330e 00007fe12a6f21ee 00007fe12a6f8425 00007fe12a6f8d4f 00007fe12a6f8b78 0000000000a71ecc
0110 20:52:20.133225 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39290112}
W20250110 20:52:20.135668 3089 consensus_peers.cc:487] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 -> Peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Couldn't send request to peer 730f5dd0bdf9484f805e503c0e4754f7. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:52:20.137054 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7"
I20250110 20:52:20.137418 3024 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:20.137743 3233 consensus_queue.cc:1035] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:52:20.142202 3024 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372d84f3d6134989b052a99839ab6a65 in term 1.
W20250110 20:52:20.143616 3024 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.3:51333 (request call id 4) took 6 ms (client timeout 7 ms). Trace:
W20250110 20:52:20.143860 3024 rpcz_store.cc:269] 0110 20:52:20.136831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:20.136888 (+ 57us) service_pool.cc:225] Handling call
0110 20:52:20.143571 (+ 6683us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250110 20:52:20.144305 3089 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Timed out: RequestConsensusVote RPC to 127.0.104.2:44259 timed out after 0.007s (SENT)
I20250110 20:52:20.146164 3232 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:20.146363 3232 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.151224 3069 heartbeater.cc:502] Master 127.0.104.62:34885 was elected leader, sending a full tablet report...
I20250110 20:52:20.151984 3220 ts_tablet_manager.cc:1428] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7: Time spent starting tablet: real 0.081s user 0.006s sys 0.006s
W20250110 20:52:20.157989 3223 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:52:20.158529 3223 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250110 20:52:20.163431 3232 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
W20250110 20:52:20.164247 2828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:44259, user_credentials={real_user=slave}} blocked reactor thread for 59824.6us
I20250110 20:52:20.165992 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
W20250110 20:52:20.167686 2958 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.1:39435 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20250110 20:52:20.168658 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:20.171530 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:20.175146 3227 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:52:20.179030 2958 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Timed out: connection negotiation to 127.0.104.1:39435 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
W20250110 20:52:20.180974 2960 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505): Timed out: connection negotiation to 127.0.104.3:44505 for RPC RequestConsensusVote timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20250110 20:52:20.181129 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:20.181824 3238 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:20.185190 3227 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:20.185426 3227 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.186331 3227 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:20.187247 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:20.187779 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:20.188413 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:20.188819 3227 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:20.190522 3238 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:20.190711 3238 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.191507 3238 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:20.192380 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:20.192431 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:20.193405 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:20.193928 3238 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:20.506136 3227 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:20.506350 3227 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.507032 3227 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:20.507651 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:20.507725 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:20.508499 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:20.508824 3227 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:52:20.705104 2893 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:52:20.705229 2893 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250110 20:52:20.706594 3022 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:52:20.706683 3022 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250110 20:52:20.881995 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:20.882191 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:20.883349 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:20.883877 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:20.883888 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:20.884472 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:20.885277 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:52:20.939992 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.611s user 0.001s sys 0.000s
W20250110 20:52:20.941614 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.613s user 0.001s sys 0.000s
I20250110 20:52:21.049505 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:21.049736 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:21.050535 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:21.051286 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:21.051350 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:21.052223 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:21.052702 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:21.057845 3227 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:21.058102 3227 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:21.059237 3227 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:21.060207 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:21.060256 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:21.061292 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:21.061903 3227 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:21.168587 3227 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:21.168829 3227 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:21.169663 3227 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:21.170114 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:21.170356 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:21.170946 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:21.171356 3227 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:21.695542 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:21.695758 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:21.696588 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:21.696936 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:21.696959 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:21.697619 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:21.697923 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:21.890146 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:21.890362 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:21.891208 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:21.891639 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:21.892182 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:21.892737 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:21.893013 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:52:21.940109 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.597s user 0.001s sys 0.000s
W20250110 20:52:21.940171 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.596s user 0.001s sys 0.000s
I20250110 20:52:22.423333 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:22.423560 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:22.424459 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:22.424878 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:22.424999 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:22.425650 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:22.633018 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:22.633195 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:22.634022 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:22.634433 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:22.634523 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:22.635041 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:22.635615 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:22.658587 3269 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=3, attempt_no=0}) took 1467 ms. Trace:
I20250110 20:52:22.658799 3269 rpcz_store.cc:276] 0110 20:52:21.191600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:21.191677 (+ 77us) service_pool.cc:225] Handling call
0110 20:52:22.658529 (+1466852us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:21.723871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:21.724015 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:21.724022 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:21.724024 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:21.724140 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:21.724152 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:21.724156 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:21.724216 (+ 60us) tablet.cc:698] Row locks acquired
0110 20:52:21.724219 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:21.724277 (+ 58us) write_op.cc:270] Start()
0110 20:52:21.914443 (+190166us) write_op.cc:276] Timestamp: P: 1736542341724222 usec, L: 0
0110 20:52:21.914450 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:21.914660 (+ 210us) log.cc:844] Serialized 354 byte log entry
0110 20:52:22.657564 (+742904us) op_driver.cc:464] REPLICATION: finished
0110 20:52:22.657986 (+ 422us) write_op.cc:301] APPLY: starting
0110 20:52:22.658002 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:22.658038 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:22.658060 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:22.658150 (+ 90us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:22.658177 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:22.658182 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:22.658265 (+ 83us) log.cc:844] Serialized 22 byte log entry
0110 20:52:22.658368 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:22.658387 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:22.658393 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":355,"num_ops":1,"prepare.queue_time_us":531653,"prepare.run_cpu_time_us":791,"prepare.run_wall_time_us":190869,"replication_time_us":743057,"thread_start_us":271,"threads_started":1}]]}
I20250110 20:52:22.772807 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:23.431308 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:23.431516 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:23.432332 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:23.432806 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:23.432979 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:23.433557 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:23.433838 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:23.447378 3274 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=4, attempt_no=0}) took 1507 ms. Trace:
I20250110 20:52:23.447621 3274 rpcz_store.cc:276] 0110 20:52:21.939574 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:21.939653 (+ 79us) service_pool.cc:225] Handling call
0110 20:52:23.447306 (+1507653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:21.940218 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:21.940330 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:21.940337 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:21.940339 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:21.940430 (+ 91us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:21.940442 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:21.940447 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:21.940488 (+ 41us) tablet.cc:698] Row locks acquired
0110 20:52:21.940490 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:21.940524 (+ 34us) write_op.cc:270] Start()
0110 20:52:22.900698 (+960174us) write_op.cc:276] Timestamp: P: 1736542341940450 usec, L: 0
0110 20:52:22.900705 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:22.900985 (+ 280us) log.cc:844] Serialized 354 byte log entry
0110 20:52:23.446153 (+545168us) op_driver.cc:464] REPLICATION: finished
0110 20:52:23.446697 (+ 544us) write_op.cc:301] APPLY: starting
0110 20:52:23.446718 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:23.446781 (+ 63us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:23.446814 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:23.446909 (+ 95us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:23.446939 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:23.446944 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:23.447040 (+ 96us) log.cc:844] Serialized 22 byte log entry
0110 20:52:23.447167 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:23.447189 (+ 22us) write_op.cc:454] Released schema lock
0110 20:52:23.447196 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":427,"num_ops":1,"prepare.queue_time_us":61,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":960923,"raft.queue_time_us":39,"raft.run_cpu_time_us":33,"raft.run_wall_time_us":33,"replication_time_us":545385,"spinlock_wait_cycles":132608,"thread_start_us":286,"threads_started":1}]]}
I20250110 20:52:23.448871 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 12) took 1357 ms. Trace:
I20250110 20:52:23.448945 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 11) took 1357 ms. Trace:
I20250110 20:52:23.449076 3114 rpcz_store.cc:276] 0110 20:52:22.091623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:22.091717 (+ 94us) service_pool.cc:225] Handling call
0110 20:52:22.091989 (+ 272us) tablet_service.cc:2780] Created scanner 714c3378e61e48c29a4c584a8166a612 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 6b7a950c362849ea8ea441d770eaefb2
0110 20:52:22.092951 (+ 962us) tablet_service.cc:2920] Creating iterator
0110 20:52:22.093002 (+ 51us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:22.901598 (+808596us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:23.447203 (+545605us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1354184 microseconds
0110 20:52:23.447317 (+ 114us) tablet_service.cc:2945] Iterator created
0110 20:52:23.447837 (+ 520us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:23.447894 (+ 57us) tablet_service.cc:3010] has_more: true
0110 20:52:23.447940 (+ 46us) tablet_service.cc:3027] Continuing scan request
0110 20:52:23.448022 (+ 82us) tablet_service.cc:3091] Found scanner 714c3378e61e48c29a4c584a8166a612 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 6b7a950c362849ea8ea441d770eaefb2
0110 20:52:23.448888 (+ 866us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":24832}
I20250110 20:52:23.448988 3112 rpcz_store.cc:276] 0110 20:52:22.091716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:22.091777 (+ 61us) service_pool.cc:225] Handling call
0110 20:52:22.091989 (+ 212us) tablet_service.cc:2780] Created scanner 63b1b39132c74874b83907727e0c6d74 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 775d1be6337444d5a716e49a86d47ed9
0110 20:52:22.092951 (+ 962us) tablet_service.cc:2920] Creating iterator
0110 20:52:22.093002 (+ 51us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:22.901621 (+808619us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:23.447212 (+545591us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1354193 microseconds
0110 20:52:23.447330 (+ 118us) tablet_service.cc:2945] Iterator created
0110 20:52:23.447778 (+ 448us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:23.447804 (+ 26us) tablet_service.cc:3010] has_more: true
0110 20:52:23.447847 (+ 43us) tablet_service.cc:3027] Continuing scan request
0110 20:52:23.447946 (+ 99us) tablet_service.cc:3091] Found scanner 63b1b39132c74874b83907727e0c6d74 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 775d1be6337444d5a716e49a86d47ed9
0110 20:52:23.448814 (+ 868us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":136192}
W20250110 20:52:23.449803 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.359s user 0.000s sys 0.000s
W20250110 20:52:23.449841 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.359s user 0.000s sys 0.000s
I20250110 20:52:23.778247 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:23.778467 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:23.779258 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:23.779668 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:23.779894 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:23.780452 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:23.872398 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:23.872602 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:23.873306 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:23.873812 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:23.874001 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:23.874529 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:23.874850 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:23.906435 3274 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=5, attempt_no=0}) took 1245 ms. Trace:
I20250110 20:52:23.906594 3274 rpcz_store.cc:276] 0110 20:52:22.661240 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:22.661303 (+ 63us) service_pool.cc:225] Handling call
0110 20:52:23.906374 (+1245071us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:22.901185 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:22.901332 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:22.901339 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:22.901341 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:22.901444 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:22.901456 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:22.901461 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:22.901516 (+ 55us) tablet.cc:698] Row locks acquired
0110 20:52:22.901518 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:22.901562 (+ 44us) write_op.cc:270] Start()
0110 20:52:22.926766 (+ 25204us) write_op.cc:276] Timestamp: P: 1736542342901506 usec, L: 0
0110 20:52:22.926774 (+ 8us) op_driver.cc:348] REPLICATION: starting
0110 20:52:22.927032 (+ 258us) log.cc:844] Serialized 354 byte log entry
0110 20:52:23.905698 (+978666us) op_driver.cc:464] REPLICATION: finished
0110 20:52:23.905854 (+ 156us) write_op.cc:301] APPLY: starting
0110 20:52:23.905875 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:23.905923 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:23.905977 (+ 54us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:23.906047 (+ 70us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:23.906086 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:23.906093 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:23.906188 (+ 95us) log.cc:844] Serialized 22 byte log entry
0110 20:52:23.906270 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:23.906287 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:23.906293 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":1,"prepare.queue_time_us":239437,"prepare.run_cpu_time_us":867,"prepare.run_wall_time_us":25930,"replication_time_us":978886}]]}
I20250110 20:52:24.474081 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:24.531867 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:24.532085 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:24.532904 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:24.533346 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:24.533946 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:24.534508 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:24.534790 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:24.576141 3284 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=6, attempt_no=0}) took 1126 ms. Trace:
I20250110 20:52:24.576296 3284 rpcz_store.cc:276] 0110 20:52:23.449814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:23.449860 (+ 46us) service_pool.cc:225] Handling call
0110 20:52:24.576076 (+1126216us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:23.450690 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:23.450849 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:23.450854 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:23.450856 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:23.450928 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:23.450938 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:23.450941 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:23.450992 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:52:23.450993 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:23.451025 (+ 32us) write_op.cc:270] Start()
0110 20:52:24.213186 (+762161us) write_op.cc:276] Timestamp: P: 1736542343450956 usec, L: 0
0110 20:52:24.213192 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:24.213456 (+ 264us) log.cc:844] Serialized 354 byte log entry
0110 20:52:24.575026 (+361570us) op_driver.cc:464] REPLICATION: finished
0110 20:52:24.575408 (+ 382us) write_op.cc:301] APPLY: starting
0110 20:52:24.575425 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:24.575469 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:24.575494 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:24.575599 (+ 105us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:24.575635 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:24.575642 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:24.575765 (+ 123us) log.cc:844] Serialized 22 byte log entry
0110 20:52:24.575892 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:24.575913 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:24.575920 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":303,"num_ops":1,"prepare.queue_time_us":330,"prepare.run_cpu_time_us":762,"prepare.run_wall_time_us":762832,"replication_time_us":361795,"thread_start_us":439,"threads_started":2}]]}
W20250110 20:52:24.578366 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.978s user 0.001s sys 0.000s
W20250110 20:52:24.578894 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.978s user 0.001s sys 0.000s
I20250110 20:52:25.247157 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:25.247363 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:25.248121 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:25.248673 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:25.248683 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:25.249255 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:25.249540 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:25.257113 3291 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=7, attempt_no=0}) took 1348 ms. Trace:
I20250110 20:52:25.257259 3291 rpcz_store.cc:276] 0110 20:52:23.908300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:23.908366 (+ 66us) service_pool.cc:225] Handling call
0110 20:52:25.257059 (+1348693us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:24.213586 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:24.213712 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:24.213719 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:24.213721 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:24.213820 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:24.213834 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:24.213839 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:24.213895 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:24.213897 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:24.213942 (+ 45us) write_op.cc:270] Start()
0110 20:52:24.722081 (+508139us) write_op.cc:276] Timestamp: P: 1736542344213877 usec, L: 0
0110 20:52:24.722086 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:24.722300 (+ 214us) log.cc:844] Serialized 354 byte log entry
0110 20:52:25.256011 (+533711us) op_driver.cc:464] REPLICATION: finished
0110 20:52:25.256507 (+ 496us) write_op.cc:301] APPLY: starting
0110 20:52:25.256523 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:25.256561 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:25.256610 (+ 49us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:25.256700 (+ 90us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:25.256727 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:25.256732 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:25.256835 (+ 103us) log.cc:844] Serialized 22 byte log entry
0110 20:52:25.256927 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:25.256944 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:25.256950 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":401,"num_ops":1,"prepare.queue_time_us":304719,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":508852,"raft.queue_time_us":47,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":7,"replication_time_us":533858,"spinlock_wait_cycles":14592,"thread_start_us":281,"threads_started":1}]]}
I20250110 20:52:25.438232 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:25.438436 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:25.439234 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:25.439823 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:25.440006 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:25.440661 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:25.440953 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:25.686218 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:25.686436 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:25.687204 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:25.687618 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:25.687760 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:25.688369 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:25.697057 3291 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=8, attempt_no=0}) took 1118 ms. Trace:
I20250110 20:52:25.697219 3291 rpcz_store.cc:276] 0110 20:52:24.578435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:24.578494 (+ 59us) service_pool.cc:225] Handling call
0110 20:52:25.696951 (+1118457us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:24.722467 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:24.722611 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:24.722619 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:24.722621 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:24.722720 (+ 99us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:24.722733 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:24.722767 (+ 34us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:24.722827 (+ 60us) tablet.cc:698] Row locks acquired
0110 20:52:24.722830 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:24.722877 (+ 47us) write_op.cc:270] Start()
0110 20:52:24.839020 (+116143us) write_op.cc:276] Timestamp: P: 1736542344722838 usec, L: 0
0110 20:52:24.839026 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:24.839354 (+ 328us) log.cc:844] Serialized 354 byte log entry
0110 20:52:25.696276 (+856922us) op_driver.cc:464] REPLICATION: finished
0110 20:52:25.696405 (+ 129us) write_op.cc:301] APPLY: starting
0110 20:52:25.696427 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:25.696466 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:25.696490 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:25.696561 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:25.696590 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:25.696596 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:25.696690 (+ 94us) log.cc:844] Serialized 22 byte log entry
0110 20:52:25.696810 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:25.696831 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:25.696838 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":51,"num_ops":1,"prepare.queue_time_us":143432,"prepare.run_cpu_time_us":904,"prepare.run_wall_time_us":116963,"replication_time_us":857191}]]}
W20250110 20:52:25.699086 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.970s user 0.001s sys 0.000s
W20250110 20:52:25.699584 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.971s user 0.001s sys 0.000s
I20250110 20:52:25.796411 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:25.796624 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:25.797510 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:25.798095 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:25.798267 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:25.799011 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:25.982304 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:26.113368 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:26.280103 3251 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:26.280328 3251 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:26.281072 3251 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:26.281497 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:26.281679 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:26.282406 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:26.282663 3251 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:26.523248 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:26.523458 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:26.524168 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:26.524549 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:26.524623 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:26.525188 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:26.525775 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:27.098981 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:27.099205 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:27.099958 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:27.100364 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:27.100507 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:27.101112 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:27.101410 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:27.113215 3300 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=11, attempt_no=0}) took 1084 ms. Trace:
I20250110 20:52:27.113382 3300 rpcz_store.cc:276] 0110 20:52:26.028416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:26.028483 (+ 67us) service_pool.cc:225] Handling call
0110 20:52:27.113146 (+1084663us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:26.029030 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:26.029132 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:26.029137 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:26.029139 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:26.029216 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:26.029227 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:26.029233 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:26.029283 (+ 50us) tablet.cc:698] Row locks acquired
0110 20:52:26.029285 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:26.029327 (+ 42us) write_op.cc:270] Start()
0110 20:52:27.004477 (+975150us) write_op.cc:276] Timestamp: P: 1736542346029293 usec, L: 0
0110 20:52:27.004482 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:27.004757 (+ 275us) log.cc:844] Serialized 354 byte log entry
0110 20:52:27.112071 (+107314us) op_driver.cc:464] REPLICATION: finished
0110 20:52:27.112543 (+ 472us) write_op.cc:301] APPLY: starting
0110 20:52:27.112569 (+ 26us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:27.112606 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:27.112630 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:27.112704 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:27.112732 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:27.112737 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:27.112856 (+ 119us) log.cc:844] Serialized 22 byte log entry
0110 20:52:27.112997 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:27.113019 (+ 22us) write_op.cc:454] Released schema lock
0110 20:52:27.113026 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":399,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":733,"prepare.run_wall_time_us":975808,"replication_time_us":107524,"thread_start_us":300,"threads_started":1}]]}
W20250110 20:52:27.115482 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.672s user 0.001s sys 0.000s
W20250110 20:52:27.115550 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.672s user 0.001s sys 0.000s
I20250110 20:52:27.624123 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:27.624361 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:27.625188 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:27.625680 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:27.625699 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:27.626300 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:28.525887 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:28.526070 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:28.526815 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:28.527280 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:28.527509 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:28.528163 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:28.529445 3308 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=12, attempt_no=0}) took 2236 ms. Trace:
I20250110 20:52:28.529577 3308 rpcz_store.cc:276] 0110 20:52:26.292604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:26.292656 (+ 52us) service_pool.cc:225] Handling call
0110 20:52:28.529382 (+2236726us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:27.004886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:27.005048 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:27.005060 (+ 12us) write_op.cc:435] Acquired schema lock
0110 20:52:27.005063 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:27.005168 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:27.005181 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:27.005186 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:27.005244 (+ 58us) tablet.cc:698] Row locks acquired
0110 20:52:27.005246 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:27.005291 (+ 45us) write_op.cc:270] Start()
0110 20:52:27.578438 (+573147us) write_op.cc:276] Timestamp: P: 1736542347005257 usec, L: 0
0110 20:52:27.578444 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:27.578840 (+ 396us) log.cc:844] Serialized 354 byte log entry
0110 20:52:28.527776 (+948936us) op_driver.cc:464] REPLICATION: finished
0110 20:52:28.528865 (+ 1089us) write_op.cc:301] APPLY: starting
0110 20:52:28.528883 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:28.528926 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:28.528950 (+ 24us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:28.529022 (+ 72us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:28.529051 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:28.529057 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:28.529139 (+ 82us) log.cc:844] Serialized 22 byte log entry
0110 20:52:28.529258 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:28.529278 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:28.529285 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1016,"num_ops":1,"prepare.queue_time_us":711685,"prepare.run_cpu_time_us":977,"prepare.run_wall_time_us":574031,"replication_time_us":949294,"spinlock_wait_cycles":81664,"thread_start_us":913,"threads_started":1}]]}
I20250110 20:52:28.531167 3113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 30) took 1264 ms. Trace:
I20250110 20:52:28.531301 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 31) took 1263 ms. Trace:
I20250110 20:52:28.531270 3113 rpcz_store.cc:276] 0110 20:52:27.266881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:27.266932 (+ 51us) service_pool.cc:225] Handling call
0110 20:52:27.267107 (+ 175us) tablet_service.cc:2780] Created scanner 655e9cd0c15e452b8f89520dfb73cfcc for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 37fc31b838d44dc2aebf38761103e0fe
0110 20:52:27.267880 (+ 773us) tablet_service.cc:2920] Creating iterator
0110 20:52:27.267907 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:27.578625 (+310718us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:28.529488 (+950863us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1261568 microseconds
0110 20:52:28.529581 (+ 93us) tablet_service.cc:2945] Iterator created
0110 20:52:28.530050 (+ 469us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:28.530102 (+ 52us) tablet_service.cc:3010] has_more: true
0110 20:52:28.530148 (+ 46us) tablet_service.cc:3027] Continuing scan request
0110 20:52:28.530221 (+ 73us) tablet_service.cc:3091] Found scanner 655e9cd0c15e452b8f89520dfb73cfcc for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 37fc31b838d44dc2aebf38761103e0fe
0110 20:52:28.531107 (+ 886us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":15616}
I20250110 20:52:28.531440 3114 rpcz_store.cc:276] 0110 20:52:27.267344 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:27.267415 (+ 71us) service_pool.cc:225] Handling call
0110 20:52:27.267637 (+ 222us) tablet_service.cc:2780] Created scanner 2362a394f3944ba195610d1232e7c9cd for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 46b30100d318426f8c4b348d2025cd86
0110 20:52:27.268324 (+ 687us) tablet_service.cc:2920] Creating iterator
0110 20:52:27.268350 (+ 26us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:27.578638 (+310288us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:28.529287 (+950649us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1260922 microseconds
0110 20:52:28.529398 (+ 111us) tablet_service.cc:2945] Iterator created
0110 20:52:28.530045 (+ 647us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:28.530078 (+ 33us) tablet_service.cc:3010] has_more: true
0110 20:52:28.530125 (+ 47us) tablet_service.cc:3027] Continuing scan request
0110 20:52:28.530220 (+ 95us) tablet_service.cc:3091] Found scanner 2362a394f3944ba195610d1232e7c9cd for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 46b30100d318426f8c4b348d2025cd86
0110 20:52:28.531235 (+ 1015us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20250110 20:52:28.532110 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.266s user 0.001s sys 0.000s
W20250110 20:52:28.532147 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.265s user 0.000s sys 0.001s
I20250110 20:52:28.572849 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:28.613484 3308 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=13, attempt_no=0}) took 1497 ms. Trace:
I20250110 20:52:28.613626 3308 rpcz_store.cc:276] 0110 20:52:27.116046 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:27.116110 (+ 64us) service_pool.cc:225] Handling call
0110 20:52:28.613409 (+1497299us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:27.578959 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:27.579107 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:27.579114 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:27.579117 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:27.579218 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:27.579230 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:27.579236 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:27.579293 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:52:27.579295 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:27.579336 (+ 41us) write_op.cc:270] Start()
0110 20:52:27.697489 (+118153us) write_op.cc:276] Timestamp: P: 1736542347579301 usec, L: 0
0110 20:52:27.697494 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:27.697734 (+ 240us) log.cc:844] Serialized 354 byte log entry
0110 20:52:28.612702 (+914968us) op_driver.cc:464] REPLICATION: finished
0110 20:52:28.612878 (+ 176us) write_op.cc:301] APPLY: starting
0110 20:52:28.612904 (+ 26us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:28.612954 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:28.612982 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:28.613081 (+ 99us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:28.613111 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:28.613116 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:28.613203 (+ 87us) log.cc:844] Serialized 22 byte log entry
0110 20:52:28.613277 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:28.613296 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:28.613302 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"num_ops":1,"prepare.queue_time_us":462374,"prepare.run_cpu_time_us":799,"prepare.run_wall_time_us":118861,"replication_time_us":915132}]]}
I20250110 20:52:28.854246 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:28.854437 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:28.855160 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:28.855676 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:28.855715 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:28.856307 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:28.856631 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:29.493983 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:29.743063 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:29.743255 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:29.744020 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:29.744431 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:29.744513 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:29.745115 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:29.745405 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:29.759019 3319 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=15, attempt_no=0}) took 1143 ms. Trace:
I20250110 20:52:29.759178 3319 rpcz_store.cc:276] 0110 20:52:28.615345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:28.615418 (+ 73us) service_pool.cc:225] Handling call
0110 20:52:29.758951 (+1143533us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:28.615934 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:28.616036 (+ 102us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:28.616041 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:28.616042 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:28.616126 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:28.616136 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:28.616139 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:28.616188 (+ 49us) tablet.cc:698] Row locks acquired
0110 20:52:28.616189 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:28.616222 (+ 33us) write_op.cc:270] Start()
0110 20:52:29.219392 (+603170us) write_op.cc:276] Timestamp: P: 1736542348616193 usec, L: 0
0110 20:52:29.219399 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:29.219669 (+ 270us) log.cc:844] Serialized 354 byte log entry
0110 20:52:29.757610 (+537941us) op_driver.cc:464] REPLICATION: finished
0110 20:52:29.758080 (+ 470us) write_op.cc:301] APPLY: starting
0110 20:52:29.758101 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:29.758140 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:29.758161 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:29.758260 (+ 99us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:29.758303 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:29.758311 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:29.758443 (+ 132us) log.cc:844] Serialized 22 byte log entry
0110 20:52:29.758807 (+ 364us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:29.758828 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:29.758836 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":384,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":603805,"replication_time_us":538173,"thread_start_us":279,"threads_started":1}]]}
I20250110 20:52:29.760841 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 34) took 1077 ms. Trace:
I20250110 20:52:29.760882 3113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 35) took 1077 ms. Trace:
I20250110 20:52:29.760989 3113 rpcz_store.cc:276] 0110 20:52:28.683701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:28.683768 (+ 67us) service_pool.cc:225] Handling call
0110 20:52:28.684001 (+ 233us) tablet_service.cc:2780] Created scanner db8aabacc796444cb98f9e4b0cea2d23 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is de9de01af81a40f185afa4129d3c2619
0110 20:52:28.684937 (+ 936us) tablet_service.cc:2920] Creating iterator
0110 20:52:28.684987 (+ 50us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:29.219552 (+534565us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:29.758846 (+539294us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1073843 microseconds
0110 20:52:29.758953 (+ 107us) tablet_service.cc:2945] Iterator created
0110 20:52:29.759628 (+ 675us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:29.759670 (+ 42us) tablet_service.cc:3010] has_more: true
0110 20:52:29.759717 (+ 47us) tablet_service.cc:3027] Continuing scan request
0110 20:52:29.759795 (+ 78us) tablet_service.cc:3091] Found scanner db8aabacc796444cb98f9e4b0cea2d23 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is de9de01af81a40f185afa4129d3c2619
0110 20:52:29.760771 (+ 976us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":130432}
I20250110 20:52:29.760969 3114 rpcz_store.cc:276] 0110 20:52:28.683596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:28.683696 (+ 100us) service_pool.cc:225] Handling call
0110 20:52:28.684001 (+ 305us) tablet_service.cc:2780] Created scanner e371e8311e78440997791f7cc7f807d4 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is d0977232c38548b2a8c1f4f44f6e9f71
0110 20:52:28.684937 (+ 936us) tablet_service.cc:2920] Creating iterator
0110 20:52:28.684986 (+ 49us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:29.219531 (+534545us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:29.758823 (+539292us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1073822 microseconds
0110 20:52:29.758940 (+ 117us) tablet_service.cc:2945] Iterator created
0110 20:52:29.759628 (+ 688us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:29.759665 (+ 37us) tablet_service.cc:3010] has_more: true
0110 20:52:29.759717 (+ 52us) tablet_service.cc:3027] Continuing scan request
0110 20:52:29.759795 (+ 78us) tablet_service.cc:3091] Found scanner e371e8311e78440997791f7cc7f807d4 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is d0977232c38548b2a8c1f4f44f6e9f71
0110 20:52:29.760771 (+ 976us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":33664}
W20250110 20:52:29.761620 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.079s user 0.000s sys 0.000s
W20250110 20:52:29.761796 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.079s user 0.000s sys 0.000s
I20250110 20:52:30.059432 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:30.059640 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:30.060490 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:30.060930 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:30.061055 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:30.061686 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:30.245365 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:30.245573 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:30.246317 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:30.246783 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:30.246985 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:30.247527 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:30.247814 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:30.260363 3319 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=16, attempt_no=0}) took 1397 ms. Trace:
I20250110 20:52:30.260488 3319 rpcz_store.cc:276] 0110 20:52:28.862555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:28.862623 (+ 68us) service_pool.cc:225] Handling call
0110 20:52:30.260296 (+1397673us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:29.219780 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:29.219927 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:29.219933 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:29.219936 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:29.220033 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:29.220046 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:29.220050 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:29.220108 (+ 58us) tablet.cc:698] Row locks acquired
0110 20:52:29.220111 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:29.220155 (+ 44us) write_op.cc:270] Start()
0110 20:52:29.842309 (+622154us) write_op.cc:276] Timestamp: P: 1736542349220100 usec, L: 0
0110 20:52:29.842316 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:29.842591 (+ 275us) log.cc:844] Serialized 354 byte log entry
0110 20:52:30.259656 (+417065us) op_driver.cc:464] REPLICATION: finished
0110 20:52:30.259782 (+ 126us) write_op.cc:301] APPLY: starting
0110 20:52:30.259800 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:30.259840 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:30.259872 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:30.259951 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:30.259981 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:30.259987 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:30.260077 (+ 90us) log.cc:844] Serialized 22 byte log entry
0110 20:52:30.260179 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:30.260197 (+ 18us) write_op.cc:454] Released schema lock
0110 20:52:30.260202 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":49,"num_ops":1,"prepare.queue_time_us":356714,"prepare.run_cpu_time_us":820,"prepare.run_wall_time_us":622887,"replication_time_us":417288}]]}
I20250110 20:52:30.589355 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:30.589581 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:30.590440 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:30.590888 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:30.591141 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:30.591851 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
W20250110 20:52:30.618530 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.706s user 0.001s sys 0.000s
W20250110 20:52:30.618541 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.706s user 0.000s sys 0.000s
I20250110 20:52:31.540782 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:31.541229 2893 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:55115 (request call id 2143) took 1480 ms. Trace:
I20250110 20:52:31.541370 2893 rpcz_store.cc:276] 0110 20:52:30.061060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:30.061110 (+ 50us) service_pool.cc:225] Handling call
0110 20:52:30.061302 (+ 192us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:52:30.061467 (+ 165us) raft_consensus.cc:1529] Early marking committed up to index 17
0110 20:52:30.061584 (+ 117us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:52:30.061740 (+ 156us) write_op.cc:270] Start()
0110 20:52:31.009930 (+948190us) write_op.cc:276] Timestamp: P: 1736542349220100 usec, L: 0
0110 20:52:31.010566 (+ 636us) write_op.cc:270] Start()
0110 20:52:31.539737 (+529171us) write_op.cc:276] Timestamp: P: 1736542349843037 usec, L: 0
0110 20:52:31.540341 (+ 604us) log.cc:844] Serialized 708 byte log entry
0110 20:52:31.540652 (+ 311us) raft_consensus.cc:1640] Marking committed up to 17
0110 20:52:31.540662 (+ 10us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:52:31.540704 (+ 42us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:52:31.541117 (+ 413us) raft_consensus.cc:1680] finished
0110 20:52:31.541120 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:52:31.541174 (+ 54us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:31.010482 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:31.010612 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:31.010618 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:31.010619 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:31.010704 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:31.010724 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:31.010727 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:31.010806 (+ 79us) tablet.cc:698] Row locks acquired
0110 20:52:31.010807 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0110 20:52:31.540232 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:31.540346 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:31.540352 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:31.540354 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:31.540437 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:31.540450 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:31.540453 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:31.540509 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:31.540511 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":232,"threads_started":1,"wal-append.queue_time_us":313,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":387,"prepare.run_cpu_time_us":416,"prepare.run_wall_time_us":417,"thread_start_us":262,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":347,"prepare.run_cpu_time_us":380,"prepare.run_wall_time_us":380,"thread_start_us":219,"threads_started":1}]]}
I20250110 20:52:31.555620 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:31.555845 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:31.556694 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:31.557199 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:31.557266 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:31.557885 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:31.558348 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:31.558893 3331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=18, attempt_no=0}) took 1296 ms. Trace:
I20250110 20:52:31.559012 3331 rpcz_store.cc:276] 0110 20:52:30.262106 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:30.262175 (+ 69us) service_pool.cc:225] Handling call
0110 20:52:31.558834 (+1296659us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:30.262726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:30.262853 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:30.262858 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:30.262860 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:30.262938 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:30.262947 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:30.262951 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:30.262993 (+ 42us) tablet.cc:698] Row locks acquired
0110 20:52:30.262995 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:30.263030 (+ 35us) write_op.cc:270] Start()
0110 20:52:30.414221 (+151191us) write_op.cc:276] Timestamp: P: 1736542350262986 usec, L: 0
0110 20:52:30.414249 (+ 28us) op_driver.cc:348] REPLICATION: starting
0110 20:52:30.414521 (+ 272us) log.cc:844] Serialized 354 byte log entry
0110 20:52:31.557321 (+1142800us) op_driver.cc:464] REPLICATION: finished
0110 20:52:31.558188 (+ 867us) write_op.cc:301] APPLY: starting
0110 20:52:31.558226 (+ 38us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:31.558278 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:31.558336 (+ 58us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:31.558456 (+ 120us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:31.558495 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:31.558502 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:31.558621 (+ 119us) log.cc:844] Serialized 22 byte log entry
0110 20:52:31.558698 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:31.558714 (+ 16us) write_op.cc:454] Released schema lock
0110 20:52:31.558722 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":790,"num_ops":1,"prepare.queue_time_us":63,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":151881,"replication_time_us":1142995,"thread_start_us":698,"threads_started":1}]]}
I20250110 20:52:31.587910 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:31.839960 3261 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:31.840165 3261 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:31.840996 3261 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:31.841434 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:31.841645 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:31.842236 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:31.842499 3261 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:31.871088 3331 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=19, attempt_no=0}) took 1253 ms. Trace:
I20250110 20:52:31.871243 3331 rpcz_store.cc:276] 0110 20:52:30.618045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:30.618088 (+ 43us) service_pool.cc:225] Handling call
0110 20:52:31.871008 (+1252920us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:30.618656 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:30.618806 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:30.618812 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:30.618815 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:30.618900 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:30.618909 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:30.618913 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:30.618954 (+ 41us) tablet.cc:698] Row locks acquired
0110 20:52:30.618956 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:30.618989 (+ 33us) write_op.cc:270] Start()
0110 20:52:31.569118 (+950129us) write_op.cc:276] Timestamp: P: 1736542350618952 usec, L: 0
0110 20:52:31.569123 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:31.569387 (+ 264us) log.cc:844] Serialized 354 byte log entry
0110 20:52:31.870298 (+300911us) op_driver.cc:464] REPLICATION: finished
0110 20:52:31.870414 (+ 116us) write_op.cc:301] APPLY: starting
0110 20:52:31.870437 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:31.870473 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:31.870520 (+ 47us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:31.870610 (+ 90us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:31.870648 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:31.870655 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:31.870774 (+ 119us) log.cc:844] Serialized 22 byte log entry
0110 20:52:31.870883 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:31.870904 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:31.870912 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":738,"prepare.run_wall_time_us":950802,"replication_time_us":301139}]]}
I20250110 20:52:31.872817 3113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 42) took 1102 ms. Trace:
I20250110 20:52:31.872956 3113 rpcz_store.cc:276] 0110 20:52:30.770237 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:30.770334 (+ 97us) service_pool.cc:225] Handling call
0110 20:52:30.770563 (+ 229us) tablet_service.cc:2780] Created scanner 91910a8574884d82bebad1cb75f11124 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is f7ab3560f6a1498a8b744157237ed742
0110 20:52:30.771524 (+ 961us) tablet_service.cc:2920] Creating iterator
0110 20:52:30.771558 (+ 34us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:31.569256 (+797698us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:31.870938 (+301682us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1099361 microseconds
0110 20:52:31.871089 (+ 151us) tablet_service.cc:2945] Iterator created
0110 20:52:31.871643 (+ 554us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:31.871680 (+ 37us) tablet_service.cc:3010] has_more: true
0110 20:52:31.871729 (+ 49us) tablet_service.cc:3027] Continuing scan request
0110 20:52:31.871838 (+ 109us) tablet_service.cc:3091] Found scanner 91910a8574884d82bebad1cb75f11124 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is f7ab3560f6a1498a8b744157237ed742
0110 20:52:31.872754 (+ 916us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20250110 20:52:31.873703 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.104s user 0.001s sys 0.000s
I20250110 20:52:31.874351 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 43) took 1103 ms. Trace:
I20250110 20:52:31.874439 3114 rpcz_store.cc:276] 0110 20:52:30.770394 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:30.770458 (+ 64us) service_pool.cc:225] Handling call
0110 20:52:30.770633 (+ 175us) tablet_service.cc:2780] Created scanner 813e9439c6ed40f48a80e88bd53297c9 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is fa557eff63874a519c8a147e3acdc307
0110 20:52:30.771457 (+ 824us) tablet_service.cc:2920] Creating iterator
0110 20:52:30.771491 (+ 34us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:31.569268 (+797777us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:31.871038 (+301770us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1099537 microseconds
0110 20:52:31.871129 (+ 91us) tablet_service.cc:2945] Iterator created
0110 20:52:31.871716 (+ 587us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:31.871747 (+ 31us) tablet_service.cc:3010] has_more: true
0110 20:52:31.871792 (+ 45us) tablet_service.cc:3027] Continuing scan request
0110 20:52:31.871858 (+ 66us) tablet_service.cc:3091] Found scanner 813e9439c6ed40f48a80e88bd53297c9 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is fa557eff63874a519c8a147e3acdc307
0110 20:52:31.874306 (+ 2448us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20250110 20:52:31.874984 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.106s user 0.001s sys 0.000s
I20250110 20:52:32.083856 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:32.084034 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:32.084817 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:32.085167 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:32.085914 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:32.086697 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:32.087154 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:32.475395 3334 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:32.475644 3334 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:32.476456 3334 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:32.476915 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:32.476994 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:32.477594 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:32.477890 3334 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:33.164916 3338 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:33.165154 3338 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:33.165942 3338 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:33.166456 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:33.166484 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:33.167153 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:33.167685 3338 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:33.214502 3342 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=21, attempt_no=0}) took 1341 ms. Trace:
I20250110 20:52:33.214669 3342 rpcz_store.cc:276] 0110 20:52:31.873154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:31.873205 (+ 51us) service_pool.cc:225] Handling call
0110 20:52:33.214432 (+1341227us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:32.421437 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:32.421574 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:32.421582 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:32.421584 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:32.421699 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:32.421710 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:32.421716 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:32.421766 (+ 50us) tablet.cc:698] Row locks acquired
0110 20:52:32.421768 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:32.421807 (+ 39us) write_op.cc:270] Start()
0110 20:52:33.068951 (+647144us) write_op.cc:276] Timestamp: P: 1736542352421746 usec, L: 0
0110 20:52:33.068957 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:33.069269 (+ 312us) log.cc:844] Serialized 354 byte log entry
0110 20:52:33.211486 (+142217us) op_driver.cc:464] REPLICATION: finished
0110 20:52:33.211922 (+ 436us) write_op.cc:301] APPLY: starting
0110 20:52:33.211942 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:33.211986 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:33.212036 (+ 50us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:33.212122 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:33.212152 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:33.212157 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:33.212287 (+ 130us) log.cc:844] Serialized 22 byte log entry
0110 20:52:33.214256 (+ 1969us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:33.214277 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:33.214285 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":351,"num_ops":1,"prepare.queue_time_us":547679,"prepare.run_cpu_time_us":842,"prepare.run_wall_time_us":647909,"replication_time_us":142483,"thread_start_us":218,"threads_started":1}]]}
W20250110 20:52:33.216290 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.546s user 0.001s sys 0.000s
W20250110 20:52:33.216552 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.546s user 0.001s sys 0.000s
I20250110 20:52:33.219766 3338 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:33.219940 3338 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:33.220568 3338 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:33.221019 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:33.221385 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:33.221899 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:33.222173 3338 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:33.379145 3297 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:33.379350 3297 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:33.380055 3297 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:33.380535 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:33.380795 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:33.381327 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:33.441730 3297 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:34.353528 3348 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:34.353758 3348 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:34.354624 3348 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:34.355029 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:34.355149 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:34.355754 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:34.356084 3348 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:34.397010 3353 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=22, attempt_no=0}) took 1877 ms. Trace:
I20250110 20:52:34.397177 3353 rpcz_store.cc:276] 0110 20:52:32.519681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:32.519734 (+ 53us) service_pool.cc:225] Handling call
0110 20:52:34.396937 (+1877203us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:33.069386 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:33.069514 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:33.069521 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:33.069524 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:33.069621 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:33.069634 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:33.069639 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:33.069693 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:52:33.069696 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:33.069737 (+ 41us) write_op.cc:270] Start()
0110 20:52:33.980887 (+911150us) write_op.cc:276] Timestamp: P: 1736542353069713 usec, L: 0
0110 20:52:33.980893 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:33.981200 (+ 307us) log.cc:844] Serialized 354 byte log entry
0110 20:52:34.395779 (+414579us) op_driver.cc:464] REPLICATION: finished
0110 20:52:34.396245 (+ 466us) write_op.cc:301] APPLY: starting
0110 20:52:34.396263 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:34.396300 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:34.396330 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:34.396431 (+ 101us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:34.396468 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:34.396476 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:34.396600 (+ 124us) log.cc:844] Serialized 22 byte log entry
0110 20:52:34.396712 (+ 112us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:34.396733 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:34.396741 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":388,"num_ops":1,"prepare.queue_time_us":549259,"prepare.run_cpu_time_us":821,"prepare.run_wall_time_us":911892,"replication_time_us":414843,"thread_start_us":272,"threads_started":1}]]}
I20250110 20:52:34.398533 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 53) took 1030 ms. Trace:
I20250110 20:52:34.398583 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 52) took 1030 ms. Trace:
I20250110 20:52:34.398682 3112 rpcz_store.cc:276] 0110 20:52:33.367647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:33.367723 (+ 76us) service_pool.cc:225] Handling call
0110 20:52:33.367959 (+ 236us) tablet_service.cc:2780] Created scanner 06e5c0053d244d5dab0c8839859c5149 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 59e0e95f05eb41fa8f30f2f113cfc031
0110 20:52:33.368915 (+ 956us) tablet_service.cc:2920] Creating iterator
0110 20:52:33.368958 (+ 43us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:33.981020 (+612062us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:34.396748 (+415728us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1027775 microseconds
0110 20:52:34.396894 (+ 146us) tablet_service.cc:2945] Iterator created
0110 20:52:34.397415 (+ 521us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:34.397477 (+ 62us) tablet_service.cc:3010] has_more: true
0110 20:52:34.397545 (+ 68us) tablet_service.cc:3027] Continuing scan request
0110 20:52:34.397615 (+ 70us) tablet_service.cc:3091] Found scanner 06e5c0053d244d5dab0c8839859c5149 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 59e0e95f05eb41fa8f30f2f113cfc031
0110 20:52:34.398525 (+ 910us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":30592}
I20250110 20:52:34.398662 3114 rpcz_store.cc:276] 0110 20:52:33.367792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:33.367835 (+ 43us) service_pool.cc:225] Handling call
0110 20:52:33.367994 (+ 159us) tablet_service.cc:2780] Created scanner 93c2a2d653e3426e86cf3d7598695653 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 043f287b3f404606935ff6fd92226287
0110 20:52:33.368905 (+ 911us) tablet_service.cc:2920] Creating iterator
0110 20:52:33.368966 (+ 61us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:33.981728 (+612762us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:34.396758 (+415030us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1027776 microseconds
0110 20:52:34.396892 (+ 134us) tablet_service.cc:2945] Iterator created
0110 20:52:34.397411 (+ 519us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:34.397446 (+ 35us) tablet_service.cc:3010] has_more: true
0110 20:52:34.397496 (+ 50us) tablet_service.cc:3027] Continuing scan request
0110 20:52:34.397571 (+ 75us) tablet_service.cc:3091] Found scanner 93c2a2d653e3426e86cf3d7598695653 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 043f287b3f404606935ff6fd92226287
0110 20:52:34.398462 (+ 891us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":49792}
W20250110 20:52:34.399457 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.032s user 0.001s sys 0.000s
W20250110 20:52:34.399458 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.032s user 0.001s sys 0.000s
I20250110 20:52:34.511821 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:34.512033 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:34.512837 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:34.513315 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:34.513500 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:34.514003 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:34.993006 3348 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:34.993216 3348 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:34.993988 3348 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:34.994587 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:34.994591 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:34.995225 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:34.995610 3348 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:35.002522 3360 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=23, attempt_no=0}) took 1785 ms. Trace:
I20250110 20:52:35.002660 3360 rpcz_store.cc:276] 0110 20:52:33.217320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:33.217390 (+ 70us) service_pool.cc:225] Handling call
0110 20:52:35.002453 (+1785063us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:33.981331 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:33.981456 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:33.981463 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:33.981465 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:33.981566 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:33.981577 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:33.981582 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:33.981639 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:52:33.981641 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:33.981687 (+ 46us) write_op.cc:270] Start()
0110 20:52:34.387822 (+406135us) write_op.cc:276] Timestamp: P: 1736542353981619 usec, L: 0
0110 20:52:34.387826 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:52:34.388015 (+ 189us) log.cc:844] Serialized 354 byte log entry
0110 20:52:35.001554 (+613539us) op_driver.cc:464] REPLICATION: finished
0110 20:52:35.001967 (+ 413us) write_op.cc:301] APPLY: starting
0110 20:52:35.001983 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:35.002018 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:35.002043 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:35.002105 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:35.002130 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:35.002134 (+ 4us) write_op.cc:312] APPLY: finished
0110 20:52:35.002245 (+ 111us) log.cc:844] Serialized 22 byte log entry
0110 20:52:35.002325 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:35.002344 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:35.002351 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":354,"num_ops":1,"prepare.queue_time_us":763476,"prepare.run_cpu_time_us":721,"prepare.run_wall_time_us":406783,"raft.queue_time_us":39,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":219,"replication_time_us":613690,"thread_start_us":255,"threads_started":1}]]}
I20250110 20:52:35.123404 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:35.183238 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:35.183415 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:35.184377 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:35.184916 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:35.185010 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:35.185806 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:35.186076 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:35.204283 3348 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:35.204491 3348 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:35.205477 3348 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:35.206068 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:35.206776 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:35.207319 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
W20250110 20:52:35.224315 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.674s user 0.001s sys 0.000s
W20250110 20:52:35.224460 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.674s user 0.001s sys 0.000s
I20250110 20:52:36.069167 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:36.069377 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:36.070118 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:36.070781 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:36.070731 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:36.071633 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:36.071949 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:36.112212 3368 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=25, attempt_no=0}) took 1107 ms. Trace:
I20250110 20:52:36.112347 3368 rpcz_store.cc:276] 0110 20:52:35.004310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:35.004383 (+ 73us) service_pool.cc:225] Handling call
0110 20:52:36.112146 (+1107763us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:35.015881 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:35.016009 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:35.016016 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:35.016019 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:35.016138 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:35.016149 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:35.016153 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:35.016210 (+ 57us) tablet.cc:698] Row locks acquired
0110 20:52:35.016212 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:35.016251 (+ 39us) write_op.cc:270] Start()
0110 20:52:35.205382 (+189131us) write_op.cc:276] Timestamp: P: 1736542355016169 usec, L: 0
0110 20:52:35.205388 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:35.205607 (+ 219us) log.cc:844] Serialized 354 byte log entry
0110 20:52:36.110868 (+905261us) op_driver.cc:464] REPLICATION: finished
0110 20:52:36.111315 (+ 447us) write_op.cc:301] APPLY: starting
0110 20:52:36.111334 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:36.111382 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:36.111410 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:36.111514 (+ 104us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:36.111554 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:36.111561 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:36.111657 (+ 96us) log.cc:844] Serialized 22 byte log entry
0110 20:52:36.112010 (+ 353us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:36.112030 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:36.112038 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":365,"num_ops":1,"prepare.queue_time_us":10994,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":189819,"raft.queue_time_us":24,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":184,"replication_time_us":905422,"spinlock_wait_cycles":17536,"thread_start_us":273,"threads_started":1}]]}
I20250110 20:52:36.184724 3348 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:36.664769 3348 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:36.665000 3348 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:36.665886 3348 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:36.666337 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:36.666410 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:36.666994 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:36.667414 3348 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:36.716506 3373 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=26, attempt_no=0}) took 1492 ms. Trace:
I20250110 20:52:36.716640 3373 rpcz_store.cc:276] 0110 20:52:35.223798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:35.223873 (+ 75us) service_pool.cc:225] Handling call
0110 20:52:36.716435 (+1492562us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:35.224437 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:35.224544 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:35.224550 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:35.224552 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:35.224637 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:35.224648 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:35.224654 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:35.224701 (+ 47us) tablet.cc:698] Row locks acquired
0110 20:52:35.224703 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:35.224737 (+ 34us) write_op.cc:270] Start()
0110 20:52:36.169869 (+945132us) write_op.cc:276] Timestamp: P: 1736542355224699 usec, L: 0
0110 20:52:36.169873 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:52:36.170135 (+ 262us) log.cc:844] Serialized 354 byte log entry
0110 20:52:36.714580 (+544445us) op_driver.cc:464] REPLICATION: finished
0110 20:52:36.715049 (+ 469us) write_op.cc:301] APPLY: starting
0110 20:52:36.715068 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:36.715104 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:36.715129 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:36.715208 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:36.715240 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:36.715247 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:36.715377 (+ 130us) log.cc:844] Serialized 22 byte log entry
0110 20:52:36.716276 (+ 899us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:36.716292 (+ 16us) write_op.cc:454] Released schema lock
0110 20:52:36.716302 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":385,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":945773,"replication_time_us":544639,"thread_start_us":256,"threads_started":1}]]}
I20250110 20:52:36.717348 3113 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 59) took 1341 ms. Trace:
I20250110 20:52:36.717392 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 60) took 1341 ms. Trace:
I20250110 20:52:36.717523 3114 rpcz_store.cc:276] 0110 20:52:35.375888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:35.375950 (+ 62us) service_pool.cc:225] Handling call
0110 20:52:35.376139 (+ 189us) tablet_service.cc:2780] Created scanner 2e82a0777c7f47c586def92979ef865c for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 3691b2807c06424a92a77dc2d4ae9ecf
0110 20:52:35.377086 (+ 947us) tablet_service.cc:2920] Creating iterator
0110 20:52:35.377133 (+ 47us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:36.170000 (+792867us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:36.715543 (+545543us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1338393 microseconds
0110 20:52:36.715664 (+ 121us) tablet_service.cc:2945] Iterator created
0110 20:52:36.716100 (+ 436us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:36.716131 (+ 31us) tablet_service.cc:3010] has_more: true
0110 20:52:36.716181 (+ 50us) tablet_service.cc:3027] Continuing scan request
0110 20:52:36.716249 (+ 68us) tablet_service.cc:3091] Found scanner 2e82a0777c7f47c586def92979ef865c for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 3691b2807c06424a92a77dc2d4ae9ecf
0110 20:52:36.717282 (+ 1033us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":84352}
I20250110 20:52:36.717478 3113 rpcz_store.cc:276] 0110 20:52:35.375762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:35.375846 (+ 84us) service_pool.cc:225] Handling call
0110 20:52:35.376070 (+ 224us) tablet_service.cc:2780] Created scanner c269ce3255f64652bfc50bcc1d811ca1 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is a6099065c0644ecb956af7544002b97b
0110 20:52:35.377072 (+ 1002us) tablet_service.cc:2920] Creating iterator
0110 20:52:35.377140 (+ 68us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:36.170034 (+792894us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:36.715546 (+545512us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1338392 microseconds
0110 20:52:36.715648 (+ 102us) tablet_service.cc:2945] Iterator created
0110 20:52:36.716106 (+ 458us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:36.716149 (+ 43us) tablet_service.cc:3010] has_more: true
0110 20:52:36.716185 (+ 36us) tablet_service.cc:3027] Continuing scan request
0110 20:52:36.716253 (+ 68us) tablet_service.cc:3091] Found scanner c269ce3255f64652bfc50bcc1d811ca1 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is a6099065c0644ecb956af7544002b97b
0110 20:52:36.717266 (+ 1013us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":53248}
W20250110 20:52:36.718374 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.343s user 0.001s sys 0.000s
W20250110 20:52:36.718422 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.343s user 0.001s sys 0.000s
I20250110 20:52:36.732026 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:36.732219 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:36.733011 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:36.733352 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:36.734088 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:36.734865 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:36.735165 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:37.180886 3348 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:37.181090 3348 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:37.181872 3348 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:37.182377 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:37.182427 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:37.183014 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:37.205518 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:37.205729 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:37.206518 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:37.206985 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:37.206988 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:37.207547 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:37.215780 3373 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=27, attempt_no=0}) took 1101 ms. Trace:
I20250110 20:52:37.215919 3373 rpcz_store.cc:276] 0110 20:52:36.114442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:36.114501 (+ 59us) service_pool.cc:225] Handling call
0110 20:52:37.215689 (+1101188us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:36.170249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:36.170384 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:36.170391 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:36.170394 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:36.170494 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:36.170506 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:36.170511 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:36.170567 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:36.170568 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:36.170607 (+ 39us) write_op.cc:270] Start()
0110 20:52:36.736735 (+566128us) write_op.cc:276] Timestamp: P: 1736542356170585 usec, L: 0
0110 20:52:36.736740 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:36.736913 (+ 173us) log.cc:844] Serialized 354 byte log entry
0110 20:52:37.215032 (+478119us) op_driver.cc:464] REPLICATION: finished
0110 20:52:37.215164 (+ 132us) write_op.cc:301] APPLY: starting
0110 20:52:37.215185 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:37.215226 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:37.215273 (+ 47us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:37.215382 (+ 109us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:37.215412 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:37.215417 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:37.215499 (+ 82us) log.cc:844] Serialized 22 byte log entry
0110 20:52:37.215576 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:37.215594 (+ 18us) write_op.cc:454] Released schema lock
0110 20:52:37.215599 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":53,"num_ops":1,"prepare.queue_time_us":55249,"prepare.run_cpu_time_us":696,"prepare.run_wall_time_us":566761,"raft.queue_time_us":289,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":478245}]]}
I20250110 20:52:37.341068 3348 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:37.960330 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:38.763877 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:38.764092 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:38.764822 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:38.765251 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:38.765543 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:38.766150 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:38.766402 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:38.770406 3386 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=29, attempt_no=0}) took 1552 ms. Trace:
I20250110 20:52:38.770584 3386 rpcz_store.cc:276] 0110 20:52:37.217598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:37.217666 (+ 68us) service_pool.cc:225] Handling call
0110 20:52:38.770331 (+1552665us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:37.218205 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:37.218300 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:37.218304 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:52:37.218306 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:37.218384 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:37.218393 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:37.218397 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:37.218439 (+ 42us) tablet.cc:698] Row locks acquired
0110 20:52:37.218440 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:37.218474 (+ 34us) write_op.cc:270] Start()
0110 20:52:38.150615 (+932141us) write_op.cc:276] Timestamp: P: 1736542357218456 usec, L: 0
0110 20:52:38.150619 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:52:38.150957 (+ 338us) log.cc:844] Serialized 354 byte log entry
0110 20:52:38.769339 (+618382us) op_driver.cc:464] REPLICATION: finished
0110 20:52:38.769785 (+ 446us) write_op.cc:301] APPLY: starting
0110 20:52:38.769799 (+ 14us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:38.769832 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:38.769883 (+ 51us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:38.769951 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:38.769975 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:38.769980 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:38.770084 (+ 104us) log.cc:844] Serialized 22 byte log entry
0110 20:52:38.770173 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:38.770191 (+ 18us) write_op.cc:454] Released schema lock
0110 20:52:38.770197 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":369,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":763,"prepare.run_wall_time_us":932828,"replication_time_us":618671,"thread_start_us":251,"threads_started":1}]]}
I20250110 20:52:38.772035 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 68) took 1259 ms. Trace:
I20250110 20:52:38.772166 3114 rpcz_store.cc:276] 0110 20:52:37.512238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:37.512280 (+ 42us) service_pool.cc:225] Handling call
0110 20:52:37.512472 (+ 192us) tablet_service.cc:2780] Created scanner 7043cc6c23d24b7c87bd1f342ecc7581 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 6de3d25577a84f90962e500bd353305a
0110 20:52:37.513311 (+ 839us) tablet_service.cc:2920] Creating iterator
0110 20:52:37.513353 (+ 42us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:38.150828 (+637475us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:38.770339 (+619511us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1256975 microseconds
0110 20:52:38.770446 (+ 107us) tablet_service.cc:2945] Iterator created
0110 20:52:38.770976 (+ 530us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:38.771006 (+ 30us) tablet_service.cc:3010] has_more: true
0110 20:52:38.771049 (+ 43us) tablet_service.cc:3027] Continuing scan request
0110 20:52:38.771121 (+ 72us) tablet_service.cc:3091] Found scanner 7043cc6c23d24b7c87bd1f342ecc7581 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 6de3d25577a84f90962e500bd353305a
0110 20:52:38.771995 (+ 874us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":2432}
W20250110 20:52:38.772816 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.261s user 0.001s sys 0.000s
I20250110 20:52:38.772877 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 67) took 1260 ms. Trace:
I20250110 20:52:38.772984 3112 rpcz_store.cc:276] 0110 20:52:37.512094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:37.512177 (+ 83us) service_pool.cc:225] Handling call
0110 20:52:37.512395 (+ 218us) tablet_service.cc:2780] Created scanner 484a998ffaf748a6b6045f815ffedb46 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 723ff8dca436480f9ba18e2efbd3fcf7
0110 20:52:37.513293 (+ 898us) tablet_service.cc:2920] Creating iterator
0110 20:52:37.513320 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:38.150802 (+637482us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:38.770218 (+619416us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1256883 microseconds
0110 20:52:38.770335 (+ 117us) tablet_service.cc:2945] Iterator created
0110 20:52:38.770962 (+ 627us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:38.770995 (+ 33us) tablet_service.cc:3010] has_more: true
0110 20:52:38.771044 (+ 49us) tablet_service.cc:3027] Continuing scan request
0110 20:52:38.771120 (+ 76us) tablet_service.cc:3091] Found scanner 484a998ffaf748a6b6045f815ffedb46 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 723ff8dca436480f9ba18e2efbd3fcf7
0110 20:52:38.772828 (+ 1708us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":13568}
W20250110 20:52:38.773491 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.262s user 0.001s sys 0.000s
I20250110 20:52:38.958804 3381 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:38.959028 3381 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:38.959923 3381 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:38.960407 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:38.960516 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:38.961107 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:39.127455 3381 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:39.136143 3386 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=30, attempt_no=0}) took 1775 ms. Trace:
I20250110 20:52:39.136287 3386 rpcz_store.cc:276] 0110 20:52:37.360457 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:37.360530 (+ 73us) service_pool.cc:225] Handling call
0110 20:52:39.136075 (+1775545us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:38.151066 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:38.151172 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:38.151177 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:38.151179 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:38.151256 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:38.151265 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:38.151269 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:38.151323 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:52:38.151325 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:38.151368 (+ 43us) write_op.cc:270] Start()
0110 20:52:38.309511 (+158143us) write_op.cc:276] Timestamp: P: 1736542358151290 usec, L: 0
0110 20:52:38.309518 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:52:38.309741 (+ 223us) log.cc:844] Serialized 354 byte log entry
0110 20:52:39.135520 (+825779us) op_driver.cc:464] REPLICATION: finished
0110 20:52:39.135632 (+ 112us) write_op.cc:301] APPLY: starting
0110 20:52:39.135647 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:39.135681 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:39.135707 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:39.135766 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:39.135792 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:39.135796 (+ 4us) write_op.cc:312] APPLY: finished
0110 20:52:39.135866 (+ 70us) log.cc:844] Serialized 22 byte log entry
0110 20:52:39.135958 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:39.135977 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:39.135984 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60,"num_ops":1,"prepare.queue_time_us":790062,"prepare.run_cpu_time_us":740,"prepare.run_wall_time_us":158769,"replication_time_us":825965}]]}
I20250110 20:52:39.281342 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:39.281562 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:39.282308 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:39.282727 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:39.282756 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:39.283373 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:39.560410 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:52:39.588135 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.664s user 0.001s sys 0.000s
W20250110 20:52:39.588310 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.665s user 0.001s sys 0.000s
I20250110 20:52:39.673724 3381 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:39.673913 3381 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:39.674636 3381 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:39.675036 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:39.675181 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:39.675673 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:39.675976 3381 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:40.103564 3381 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:40.103778 3381 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:40.104553 3381 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:40.104943 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:40.105335 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:40.105885 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:40.106272 3381 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:40.621243 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:40.621446 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:40.622233 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:40.622555 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:40.622817 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:40.623641 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:40.623936 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:41.261234 3400 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:41.261453 3400 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:41.262212 3400 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:41.262678 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:41.262769 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:41.263401 2829 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:41.263669 3400 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:41.303618 3404 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=33, attempt_no=0}) took 1715 ms. Trace:
I20250110 20:52:41.303778 3404 rpcz_store.cc:276] 0110 20:52:39.588132 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:39.588266 (+ 134us) service_pool.cc:225] Handling call
0110 20:52:41.303555 (+1715289us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:39.899209 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:39.899321 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:39.899327 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:39.899329 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:39.899408 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:39.899420 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:39.899424 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:39.899470 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:52:39.899471 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:39.899513 (+ 42us) write_op.cc:270] Start()
0110 20:52:40.826658 (+927145us) write_op.cc:276] Timestamp: P: 1736542359899479 usec, L: 0
0110 20:52:40.826664 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:40.826966 (+ 302us) log.cc:844] Serialized 354 byte log entry
0110 20:52:41.302454 (+475488us) op_driver.cc:464] REPLICATION: finished
0110 20:52:41.302907 (+ 453us) write_op.cc:301] APPLY: starting
0110 20:52:41.302927 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:41.302973 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:41.303006 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:41.303098 (+ 92us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:41.303126 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:41.303131 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:41.303261 (+ 130us) log.cc:844] Serialized 22 byte log entry
0110 20:52:41.303407 (+ 146us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:41.303428 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:41.303436 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":364,"mutex_wait_us":38,"num_ops":1,"prepare.queue_time_us":310437,"prepare.run_cpu_time_us":4771,"prepare.run_wall_time_us":931912,"raft.queue_time_us":113,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":475748,"thread_start_us":4291,"threads_started":2}]]}
I20250110 20:52:41.305100 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 77) took 1025 ms. Trace:
I20250110 20:52:41.305151 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 78) took 1016 ms. Trace:
I20250110 20:52:41.305198 3114 rpcz_store.cc:276] 0110 20:52:40.279261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:40.279311 (+ 50us) service_pool.cc:225] Handling call
0110 20:52:40.279536 (+ 225us) tablet_service.cc:2780] Created scanner 8f82081996a841b7bbdec8c45faa9051 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is acb39e981112491fb2ef76545d219da8
0110 20:52:40.280280 (+ 744us) tablet_service.cc:2920] Creating iterator
0110 20:52:40.280309 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:40.827193 (+546884us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:41.303457 (+476264us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1023131 microseconds
0110 20:52:41.303564 (+ 107us) tablet_service.cc:2945] Iterator created
0110 20:52:41.304068 (+ 504us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:41.304095 (+ 27us) tablet_service.cc:3010] has_more: true
0110 20:52:41.304137 (+ 42us) tablet_service.cc:3027] Continuing scan request
0110 20:52:41.304198 (+ 61us) tablet_service.cc:3091] Found scanner 8f82081996a841b7bbdec8c45faa9051 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is acb39e981112491fb2ef76545d219da8
0110 20:52:41.305052 (+ 854us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
I20250110 20:52:41.305286 3112 rpcz_store.cc:276] 0110 20:52:40.288454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:40.288513 (+ 59us) service_pool.cc:225] Handling call
0110 20:52:40.288696 (+ 183us) tablet_service.cc:2780] Created scanner 069f7836a68d4ec4a957bfdf537a4810 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 09f42fdd64fa4c15a2b717fe521914a7
0110 20:52:40.289453 (+ 757us) tablet_service.cc:2920] Creating iterator
0110 20:52:40.289484 (+ 31us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:40.827225 (+537741us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:41.303483 (+476258us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1013987 microseconds
0110 20:52:41.303572 (+ 89us) tablet_service.cc:2945] Iterator created
0110 20:52:41.304124 (+ 552us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:41.304159 (+ 35us) tablet_service.cc:3010] has_more: true
0110 20:52:41.304200 (+ 41us) tablet_service.cc:3027] Continuing scan request
0110 20:52:41.304267 (+ 67us) tablet_service.cc:3091] Found scanner 069f7836a68d4ec4a957bfdf537a4810 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 09f42fdd64fa4c15a2b717fe521914a7
0110 20:52:41.305107 (+ 840us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20250110 20:52:41.306138 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.018s user 0.001s sys 0.000s
W20250110 20:52:41.306191 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.028s user 0.001s sys 0.000s
I20250110 20:52:41.460840 3400 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:41.461047 3400 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:41.461786 3400 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:41.462224 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:41.462296 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:41.462949 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:41.463379 3400 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:41.475479 3404 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=34, attempt_no=0}) took 1346 ms. Trace:
I20250110 20:52:41.475628 3404 rpcz_store.cc:276] 0110 20:52:40.128556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:40.128627 (+ 71us) service_pool.cc:225] Handling call
0110 20:52:41.475405 (+1346778us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:40.831164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:40.831289 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:40.831297 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:40.831299 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:40.831397 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:40.831408 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:40.831413 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:40.831465 (+ 52us) tablet.cc:698] Row locks acquired
0110 20:52:40.831467 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:40.831516 (+ 49us) write_op.cc:270] Start()
0110 20:52:41.331674 (+500158us) write_op.cc:276] Timestamp: P: 1736542360831482 usec, L: 0
0110 20:52:41.331680 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:41.331891 (+ 211us) log.cc:844] Serialized 354 byte log entry
0110 20:52:41.474803 (+142912us) op_driver.cc:464] REPLICATION: finished
0110 20:52:41.474916 (+ 113us) write_op.cc:301] APPLY: starting
0110 20:52:41.474936 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:41.474981 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:41.475015 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:41.475093 (+ 78us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:41.475123 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:41.475128 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:41.475201 (+ 73us) log.cc:844] Serialized 22 byte log entry
0110 20:52:41.475286 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:41.475302 (+ 16us) write_op.cc:454] Released schema lock
0110 20:52:41.475308 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":44,"num_ops":1,"prepare.queue_time_us":702018,"prepare.run_cpu_time_us":738,"prepare.run_wall_time_us":500803,"replication_time_us":143086}]]}
I20250110 20:52:41.503484 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:41.503679 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:41.504410 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:41.504770 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:41.509876 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:41.510841 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:41.511118 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:42.192492 3400 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:42.192718 3400 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:42.193609 3400 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:42.194115 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:42.194125 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:42.194713 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:42.195004 3400 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:52:42.203657 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.747s user 0.001s sys 0.000s
W20250110 20:52:42.203657 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.747s user 0.001s sys 0.000s
I20250110 20:52:42.294574 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:42.294801 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:42.295655 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:42.296204 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:42.296489 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:42.297048 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:42.404969 3400 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:42.405159 3400 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:42.405959 3400 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:42.406359 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:42.406430 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:42.406984 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:42.407393 3400 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:42.980238 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:43.814129 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:43.814424 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:43.815307 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:43.815764 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:43.815896 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 38 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:43.816437 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:44.223522 3418 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:44.223728 3418 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:44.224661 3418 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:44.225098 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:44.225270 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:44.225817 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:44.228200 3426 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=37, attempt_no=0}) took 2024 ms. Trace:
I20250110 20:52:44.228331 3426 rpcz_store.cc:276] 0110 20:52:42.204081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:42.204142 (+ 61us) service_pool.cc:225] Handling call
0110 20:52:44.228121 (+2023979us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:42.310826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:42.310953 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:42.310958 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:42.310960 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:42.311041 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:42.311054 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:42.311058 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:42.311114 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:42.311116 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:42.311170 (+ 54us) write_op.cc:270] Start()
0110 20:52:43.302313 (+991143us) write_op.cc:276] Timestamp: P: 1736542362311137 usec, L: 0
0110 20:52:43.302318 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:43.302600 (+ 282us) log.cc:844] Serialized 354 byte log entry
0110 20:52:44.227178 (+924578us) op_driver.cc:464] REPLICATION: finished
0110 20:52:44.227602 (+ 424us) write_op.cc:301] APPLY: starting
0110 20:52:44.227622 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:44.227663 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:44.227685 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:44.227783 (+ 98us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:44.227809 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:44.227814 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:44.227886 (+ 72us) log.cc:844] Serialized 22 byte log entry
0110 20:52:44.228003 (+ 117us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:44.228022 (+ 19us) write_op.cc:454] Released schema lock
0110 20:52:44.228028 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":340,"num_ops":1,"prepare.queue_time_us":106192,"prepare.run_cpu_time_us":783,"prepare.run_wall_time_us":991846,"replication_time_us":924828,"spinlock_wait_cycles":7424,"thread_start_us":245,"threads_started":1}]]}
I20250110 20:52:44.229780 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 84) took 1874 ms. Trace:
I20250110 20:52:44.229900 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 85) took 1874 ms. Trace:
I20250110 20:52:44.230021 3114 rpcz_store.cc:276] 0110 20:52:42.355396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:42.355462 (+ 66us) service_pool.cc:225] Handling call
0110 20:52:42.355630 (+ 168us) tablet_service.cc:2780] Created scanner 2d7ca351ae4a46ff94309e289576475c for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is fde7972d0ba94d2dbe7f56fc4febad73
0110 20:52:42.356538 (+ 908us) tablet_service.cc:2920] Creating iterator
0110 20:52:42.356590 (+ 52us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:43.302459 (+945869us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:44.228045 (+925586us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1871440 microseconds
0110 20:52:44.228155 (+ 110us) tablet_service.cc:2945] Iterator created
0110 20:52:44.228803 (+ 648us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:44.228837 (+ 34us) tablet_service.cc:3010] has_more: true
0110 20:52:44.228880 (+ 43us) tablet_service.cc:3027] Continuing scan request
0110 20:52:44.228982 (+ 102us) tablet_service.cc:3091] Found scanner 2d7ca351ae4a46ff94309e289576475c for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is fde7972d0ba94d2dbe7f56fc4febad73
0110 20:52:44.229842 (+ 860us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":20608}
I20250110 20:52:44.229888 3112 rpcz_store.cc:276] 0110 20:52:42.355270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:42.355354 (+ 84us) service_pool.cc:225] Handling call
0110 20:52:42.355601 (+ 247us) tablet_service.cc:2780] Created scanner e5f08fa7de8b42b29bedff4e37165e31 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 9ec9156b82b841ef94474b9e2ecb7248
0110 20:52:42.356524 (+ 923us) tablet_service.cc:2920] Creating iterator
0110 20:52:42.356597 (+ 73us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:43.302465 (+945868us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:44.228051 (+925586us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1871439 microseconds
0110 20:52:44.228154 (+ 103us) tablet_service.cc:2945] Iterator created
0110 20:52:44.228663 (+ 509us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:44.228691 (+ 28us) tablet_service.cc:3010] has_more: true
0110 20:52:44.228767 (+ 76us) tablet_service.cc:3027] Continuing scan request
0110 20:52:44.228886 (+ 119us) tablet_service.cc:3091] Found scanner e5f08fa7de8b42b29bedff4e37165e31 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 9ec9156b82b841ef94474b9e2ecb7248
0110 20:52:44.229735 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":50048}
W20250110 20:52:44.230702 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.876s user 0.001s sys 0.000s
W20250110 20:52:44.231247 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.877s user 0.001s sys 0.000s
I20250110 20:52:44.247140 3418 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:44.293815 3426 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=38, attempt_no=0}) took 1874 ms. Trace:
I20250110 20:52:44.293944 3426 rpcz_store.cc:276] 0110 20:52:42.419096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:42.419157 (+ 61us) service_pool.cc:225] Handling call
0110 20:52:44.293745 (+1874588us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:43.302708 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:43.302854 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:43.302862 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:52:43.302864 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:43.302962 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:43.302976 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:43.303002 (+ 26us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:43.303061 (+ 59us) tablet.cc:698] Row locks acquired
0110 20:52:43.303062 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:43.303098 (+ 36us) write_op.cc:270] Start()
0110 20:52:43.666274 (+363176us) write_op.cc:276] Timestamp: P: 1736542363303078 usec, L: 0
0110 20:52:43.666280 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:43.666517 (+ 237us) log.cc:844] Serialized 354 byte log entry
0110 20:52:44.293155 (+626638us) op_driver.cc:464] REPLICATION: finished
0110 20:52:44.293254 (+ 99us) write_op.cc:301] APPLY: starting
0110 20:52:44.293273 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:44.293311 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:44.293331 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:44.293396 (+ 65us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:44.293434 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:44.293441 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:44.293534 (+ 93us) log.cc:844] Serialized 22 byte log entry
0110 20:52:44.293616 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:44.293636 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:44.293643 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37,"num_ops":1,"prepare.queue_time_us":882993,"prepare.run_cpu_time_us":856,"prepare.run_wall_time_us":363930,"replication_time_us":626838}]]}
I20250110 20:52:45.065308 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:45.066089 3022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:51333 (request call id 2989) took 1250 ms. Trace:
I20250110 20:52:45.066184 3022 rpcz_store.cc:276] 0110 20:52:43.816001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:43.816071 (+ 70us) service_pool.cc:225] Handling call
0110 20:52:43.816266 (+ 195us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:52:43.816386 (+ 120us) raft_consensus.cc:1529] Early marking committed up to index 0
0110 20:52:43.816389 (+ 3us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:52:43.816501 (+ 112us) write_op.cc:270] Start()
0110 20:52:44.594667 (+778166us) write_op.cc:276] Timestamp: P: 1736542362311137 usec, L: 0
0110 20:52:44.595270 (+ 603us) write_op.cc:270] Start()
0110 20:52:45.064461 (+469191us) write_op.cc:276] Timestamp: P: 1736542363303078 usec, L: 0
0110 20:52:45.064878 (+ 417us) log.cc:844] Serialized 708 byte log entry
0110 20:52:45.065183 (+ 305us) raft_consensus.cc:1640] Marking committed up to 38
0110 20:52:45.065190 (+ 7us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:52:45.065234 (+ 44us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:52:45.065980 (+ 746us) raft_consensus.cc:1680] finished
0110 20:52:45.065983 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:52:45.066034 (+ 51us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:44.595196 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:44.595324 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:44.595330 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:44.595332 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:44.595412 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:44.595425 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:44.595428 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:44.595473 (+ 45us) tablet.cc:698] Row locks acquired
0110 20:52:44.595474 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0110 20:52:45.064742 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:45.064853 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:45.064859 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:45.064861 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:45.064953 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:45.064973 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:45.064977 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:45.065031 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:52:45.065033 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":237,"threads_started":1,"wal-append.queue_time_us":347,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":353,"prepare.run_cpu_time_us":372,"prepare.run_wall_time_us":371,"thread_start_us":247,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":113,"prepare.run_cpu_time_us":407,"prepare.run_wall_time_us":408}]]}
I20250110 20:52:45.593914 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:45.594157 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:45.595027 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:45.595444 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:45.595667 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:45.596321 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:45.596617 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:45.627868 3440 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=39, attempt_no=0}) took 1396 ms. Trace:
I20250110 20:52:45.628038 3440 rpcz_store.cc:276] 0110 20:52:44.230892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:44.230963 (+ 71us) service_pool.cc:225] Handling call
0110 20:52:45.627798 (+1396835us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:44.231815 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:44.231952 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:44.231957 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:44.231959 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:44.232041 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:44.232050 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:44.232053 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:44.232104 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:52:44.232105 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:44.232142 (+ 37us) write_op.cc:270] Start()
0110 20:52:44.860308 (+628166us) write_op.cc:276] Timestamp: P: 1736542364232076 usec, L: 0
0110 20:52:44.860313 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:44.860542 (+ 229us) log.cc:844] Serialized 354 byte log entry
0110 20:52:45.626703 (+766161us) op_driver.cc:464] REPLICATION: finished
0110 20:52:45.627228 (+ 525us) write_op.cc:301] APPLY: starting
0110 20:52:45.627246 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:45.627285 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:45.627320 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:45.627392 (+ 72us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:45.627418 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:45.627423 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:45.627529 (+ 106us) log.cc:844] Serialized 22 byte log entry
0110 20:52:45.627652 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:45.627674 (+ 22us) write_op.cc:454] Released schema lock
0110 20:52:45.627681 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":433,"num_ops":1,"prepare.queue_time_us":361,"prepare.run_cpu_time_us":1008,"prepare.run_wall_time_us":629084,"raft.queue_time_us":173,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":766344,"thread_start_us":796,"threads_started":3}]]}
I20250110 20:52:45.629531 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 89) took 1247 ms. Trace:
I20250110 20:52:45.629607 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 90) took 1247 ms. Trace:
I20250110 20:52:45.629727 3114 rpcz_store.cc:276] 0110 20:52:44.382323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:44.382387 (+ 64us) service_pool.cc:225] Handling call
0110 20:52:44.382548 (+ 161us) tablet_service.cc:2780] Created scanner 402427343478446aa836e86c126745f8 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 191352cd23014765b51a2b80799d42f2
0110 20:52:44.383336 (+ 788us) tablet_service.cc:2920] Creating iterator
0110 20:52:44.383366 (+ 30us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:44.860442 (+477076us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:45.627701 (+767259us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1244322 microseconds
0110 20:52:45.627822 (+ 121us) tablet_service.cc:2945] Iterator created
0110 20:52:45.628410 (+ 588us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:45.628444 (+ 34us) tablet_service.cc:3010] has_more: true
0110 20:52:45.628499 (+ 55us) tablet_service.cc:3027] Continuing scan request
0110 20:52:45.628568 (+ 69us) tablet_service.cc:3091] Found scanner 402427343478446aa836e86c126745f8 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 191352cd23014765b51a2b80799d42f2
0110 20:52:45.629551 (+ 983us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":25344}
I20250110 20:52:45.629658 3112 rpcz_store.cc:276] 0110 20:52:44.382011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:44.382077 (+ 66us) service_pool.cc:225] Handling call
0110 20:52:44.382298 (+ 221us) tablet_service.cc:2780] Created scanner 7b9433681c9c43248079a629b60d793a for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is ca20eeda917648e1a6b4d78c9352bf93
0110 20:52:44.383115 (+ 817us) tablet_service.cc:2920] Creating iterator
0110 20:52:44.383144 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:44.860425 (+477281us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:45.627688 (+767263us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1244528 microseconds
0110 20:52:45.627823 (+ 135us) tablet_service.cc:2945] Iterator created
0110 20:52:45.628428 (+ 605us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:45.628453 (+ 25us) tablet_service.cc:3010] has_more: true
0110 20:52:45.628496 (+ 43us) tablet_service.cc:3027] Continuing scan request
0110 20:52:45.628563 (+ 67us) tablet_service.cc:3091] Found scanner 7b9433681c9c43248079a629b60d793a for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is ca20eeda917648e1a6b4d78c9352bf93
0110 20:52:45.629465 (+ 902us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":768}
W20250110 20:52:45.630467 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.249s user 0.000s sys 0.000s
W20250110 20:52:45.630456 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.249s user 0.001s sys 0.000s
I20250110 20:52:46.053602 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:46.053817 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:46.054587 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:46.055055 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:46.055142 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:46.055758 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:46.082729 3440 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=40, attempt_no=0}) took 1787 ms. Trace:
I20250110 20:52:46.082947 3440 rpcz_store.cc:276] 0110 20:52:44.295563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:44.295618 (+ 55us) service_pool.cc:225] Handling call
0110 20:52:46.082663 (+1787045us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:44.860969 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:44.861081 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:44.861088 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:44.861090 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:44.861190 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:44.861201 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:44.861207 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:44.861259 (+ 52us) tablet.cc:698] Row locks acquired
0110 20:52:44.861260 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:44.861299 (+ 39us) write_op.cc:270] Start()
0110 20:52:44.958469 (+ 97170us) write_op.cc:276] Timestamp: P: 1736542364861270 usec, L: 0
0110 20:52:44.958477 (+ 8us) op_driver.cc:348] REPLICATION: starting
0110 20:52:44.958772 (+ 295us) log.cc:844] Serialized 354 byte log entry
0110 20:52:46.082026 (+1123254us) op_driver.cc:464] REPLICATION: finished
0110 20:52:46.082185 (+ 159us) write_op.cc:301] APPLY: starting
0110 20:52:46.082207 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:46.082247 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:46.082280 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:46.082355 (+ 75us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:46.082383 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:46.082388 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:46.082467 (+ 79us) log.cc:844] Serialized 22 byte log entry
0110 20:52:46.082545 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:46.082562 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:46.082568 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72,"num_ops":1,"prepare.queue_time_us":564832,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":97878,"replication_time_us":1123504}]]}
I20250110 20:52:46.346004 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:46.346230 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:46.346611 3022 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:51333 (request call id 2990) took 1278 ms. Trace:
I20250110 20:52:46.347038 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:46.346710 3022 rpcz_store.cc:276] 0110 20:52:45.067863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:45.067923 (+ 60us) service_pool.cc:225] Handling call
0110 20:52:45.068124 (+ 201us) raft_consensus.cc:1466] Updating replica for 2 ops
0110 20:52:45.068253 (+ 129us) raft_consensus.cc:1529] Early marking committed up to index 40
0110 20:52:45.068827 (+ 574us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0110 20:52:45.068998 (+ 171us) write_op.cc:270] Start()
0110 20:52:45.605169 (+536171us) write_op.cc:276] Timestamp: P: 1736542364232076 usec, L: 0
0110 20:52:45.605695 (+ 526us) write_op.cc:270] Start()
0110 20:52:46.344856 (+739161us) write_op.cc:276] Timestamp: P: 1736542364861270 usec, L: 0
0110 20:52:46.345568 (+ 712us) log.cc:844] Serialized 708 byte log entry
0110 20:52:46.345870 (+ 302us) raft_consensus.cc:1640] Marking committed up to 40
0110 20:52:46.345880 (+ 10us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:52:46.345918 (+ 38us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:52:46.346496 (+ 578us) raft_consensus.cc:1680] finished
0110 20:52:46.346498 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:52:46.346551 (+ 53us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:45.605663 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:45.605787 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:45.605793 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:45.605795 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:45.605875 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:45.605888 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:45.605892 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:45.605938 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:52:45.605940 (+ 2us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0110 20:52:46.345386 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:46.345534 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:46.345541 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:46.345544 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:46.345645 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:46.345661 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:46.345666 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:46.345763 (+ 97us) tablet.cc:698] Row locks acquired
0110 20:52:46.345766 (+ 3us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":195,"threads_started":1,"wal-append.queue_time_us":325,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":315,"prepare.run_cpu_time_us":386,"prepare.run_wall_time_us":384,"thread_start_us":204,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":384,"prepare.run_cpu_time_us":486,"prepare.run_wall_time_us":486,"thread_start_us":267,"threads_started":1}]]}
I20250110 20:52:46.347594 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:46.347587 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:46.348134 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:46.348397 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:46.381534 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:52:46.427465 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.646s user 0.001s sys 0.000s
W20250110 20:52:46.428112 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.647s user 0.001s sys 0.000s
I20250110 20:52:46.512470 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:46.512676 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:46.513381 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:46.513765 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:46.513886 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:46.514441 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:46.514683 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:47.087600 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:47.087810 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:47.088634 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:47.089075 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:47.089241 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:47.089795 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:47.378947 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:47.391539 3449 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=42, attempt_no=0}) took 1307 ms. Trace:
I20250110 20:52:47.391767 3449 rpcz_store.cc:276] 0110 20:52:46.084516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:46.084579 (+ 63us) service_pool.cc:225] Handling call
0110 20:52:47.391468 (+1306889us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:46.085081 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:46.085226 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:46.085231 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:46.085234 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:46.085307 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:46.085316 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:46.085320 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:46.085359 (+ 39us) tablet.cc:698] Row locks acquired
0110 20:52:46.085360 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:46.085393 (+ 33us) write_op.cc:270] Start()
0110 20:52:46.902524 (+817131us) write_op.cc:276] Timestamp: P: 1736542366085357 usec, L: 0
0110 20:52:46.902529 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:46.902808 (+ 279us) log.cc:844] Serialized 354 byte log entry
0110 20:52:47.390307 (+487499us) op_driver.cc:464] REPLICATION: finished
0110 20:52:47.390785 (+ 478us) write_op.cc:301] APPLY: starting
0110 20:52:47.390807 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:47.390849 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:47.390905 (+ 56us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:47.391026 (+ 121us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:47.391068 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:47.391076 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:47.391180 (+ 104us) log.cc:844] Serialized 22 byte log entry
0110 20:52:47.391334 (+ 154us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:47.391357 (+ 23us) write_op.cc:454] Released schema lock
0110 20:52:47.391364 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":360,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":760,"prepare.run_wall_time_us":817822,"replication_time_us":487718,"thread_start_us":251,"threads_started":1}]]}
W20250110 20:52:47.394214 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.816s user 0.001s sys 0.000s
W20250110 20:52:47.394287 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.816s user 0.000s sys 0.000s
I20250110 20:52:47.590160 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:47.590323 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:47.591097 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:47.591542 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:47.591943 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:47.592458 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:47.592908 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:48.036758 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:48.037077 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:48.038003 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:48.038606 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:48.038925 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:48.039450 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:48.039920 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:48.046049 3454 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=43, attempt_no=0}) took 1618 ms. Trace:
I20250110 20:52:48.046270 3454 rpcz_store.cc:276] 0110 20:52:46.428023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:46.428093 (+ 70us) service_pool.cc:225] Handling call
0110 20:52:48.045976 (+1617883us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:46.902949 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:46.903073 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:46.903080 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:46.903082 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:46.903183 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:46.903196 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:46.903200 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:46.903258 (+ 58us) tablet.cc:698] Row locks acquired
0110 20:52:46.903260 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:46.903307 (+ 47us) write_op.cc:270] Start()
0110 20:52:47.602459 (+699152us) write_op.cc:276] Timestamp: P: 1736542366903275 usec, L: 0
0110 20:52:47.602465 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:47.602778 (+ 313us) log.cc:844] Serialized 354 byte log entry
0110 20:52:48.044989 (+442211us) op_driver.cc:464] REPLICATION: finished
0110 20:52:48.045419 (+ 430us) write_op.cc:301] APPLY: starting
0110 20:52:48.045436 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:48.045482 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:48.045517 (+ 35us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:48.045597 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:48.045635 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:48.045642 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:52:48.045727 (+ 85us) log.cc:844] Serialized 22 byte log entry
0110 20:52:48.045848 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:48.045868 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:48.045876 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":366,"mutex_wait_us":35,"num_ops":1,"prepare.queue_time_us":474388,"prepare.run_cpu_time_us":904,"prepare.run_wall_time_us":699980,"raft.queue_time_us":98,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":442462,"spinlock_wait_cycles":16000,"thread_start_us":248,"threads_started":1}]]}
W20250110 20:52:48.048584 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.504s user 0.001s sys 0.000s
W20250110 20:52:48.048784 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.504s user 0.001s sys 0.000s
I20250110 20:52:48.299677 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:48.299889 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:48.300666 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:48.301098 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:48.301177 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:48.301753 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:48.302049 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:48.345168 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:48.345382 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:48.346179 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:48.346488 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:48.346774 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:48.347302 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:48.547168 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:48.547369 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:48.548228 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:48.548669 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:48.548715 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:48.549243 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:48.549695 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:48.960058 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:49.279515 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:49.279721 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:49.280440 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:49.280866 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:49.280978 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 47 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:49.281533 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:49.418496 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:49.418700 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:49.419446 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:49.419943 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:49.420020 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:49.420594 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:49.421011 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:49.445381 3463 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=46, attempt_no=0}) took 1093 ms. Trace:
I20250110 20:52:49.445551 3463 rpcz_store.cc:276] 0110 20:52:48.351566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:48.351621 (+ 55us) service_pool.cc:225] Handling call
0110 20:52:49.445309 (+1093688us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:48.472721 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:48.472841 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:48.472848 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:48.472850 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:48.472951 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:48.472964 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:48.472968 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:48.473024 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:52:48.473026 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:48.473068 (+ 42us) write_op.cc:270] Start()
0110 20:52:48.975201 (+502133us) write_op.cc:276] Timestamp: P: 1736542368473029 usec, L: 0
0110 20:52:48.975207 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:48.975461 (+ 254us) log.cc:844] Serialized 354 byte log entry
0110 20:52:49.444010 (+468549us) op_driver.cc:464] REPLICATION: finished
0110 20:52:49.444498 (+ 488us) write_op.cc:301] APPLY: starting
0110 20:52:49.444517 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:49.444566 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:49.444641 (+ 75us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:49.444734 (+ 93us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:49.444787 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:49.444804 (+ 17us) write_op.cc:312] APPLY: finished
0110 20:52:49.444936 (+ 132us) log.cc:844] Serialized 22 byte log entry
0110 20:52:49.445061 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:49.445084 (+ 23us) write_op.cc:454] Released schema lock
0110 20:52:49.445092 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":404,"num_ops":1,"prepare.queue_time_us":120636,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":502811,"replication_time_us":468756,"thread_start_us":293,"threads_started":1}]]}
W20250110 20:52:49.447547 3246 scanner-internal.cc:458] Time spent opening tablet: real 0.703s user 0.001s sys 0.000s
W20250110 20:52:49.448004 3243 scanner-internal.cc:458] Time spent opening tablet: real 0.703s user 0.001s sys 0.000s
I20250110 20:52:49.497187 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:50.154417 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:50.154664 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:50.155453 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:50.155953 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:50.155984 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:50.156658 2960 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:50.184226 3469 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=47, attempt_no=0}) took 1591 ms. Trace:
I20250110 20:52:50.184350 3469 rpcz_store.cc:276] 0110 20:52:48.593204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:48.593266 (+ 62us) service_pool.cc:225] Handling call
0110 20:52:50.184155 (+1590889us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:48.975570 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:48.975706 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:48.975712 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:48.975715 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:48.975816 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:48.975829 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:48.975834 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:48.975893 (+ 59us) tablet.cc:698] Row locks acquired
0110 20:52:48.975896 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:48.975952 (+ 56us) write_op.cc:270] Start()
0110 20:52:49.425145 (+449193us) write_op.cc:276] Timestamp: P: 1736542368975927 usec, L: 0
0110 20:52:49.425151 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:49.425369 (+ 218us) log.cc:844] Serialized 354 byte log entry
0110 20:52:50.183118 (+757749us) op_driver.cc:464] REPLICATION: finished
0110 20:52:50.183641 (+ 523us) write_op.cc:301] APPLY: starting
0110 20:52:50.183658 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:50.183697 (+ 39us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:50.183725 (+ 28us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:50.183803 (+ 78us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:50.183839 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:50.183844 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:50.183934 (+ 90us) log.cc:844] Serialized 22 byte log entry
0110 20:52:50.184016 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:50.184036 (+ 20us) write_op.cc:454] Released schema lock
0110 20:52:50.184043 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":433,"num_ops":1,"prepare.queue_time_us":381800,"prepare.run_cpu_time_us":812,"prepare.run_wall_time_us":449878,"replication_time_us":757894,"thread_start_us":320,"threads_started":1}]]}
I20250110 20:52:50.268114 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:50.268312 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:50.269143 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:50.269558 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:50.269932 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 49 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:50.270572 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:50.699753 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:50.725193 3473 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60872 (ReqId={client: f488814b900d48babbb308a61a6d976a, seq_no=48, attempt_no=0}) took 1277 ms. Trace:
I20250110 20:52:50.725337 3473 rpcz_store.cc:276] 0110 20:52:49.448093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:49.448178 (+ 85us) service_pool.cc:225] Handling call
0110 20:52:50.725130 (+1276952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:49.448683 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 257e9d67bed34c4990d8d399fde6e9e3
0110 20:52:49.448791 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:49.448796 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:49.448797 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:49.448891 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:49.448900 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:49.448904 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:49.448947 (+ 43us) tablet.cc:698] Row locks acquired
0110 20:52:49.448949 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:49.448982 (+ 33us) write_op.cc:270] Start()
0110 20:52:50.078154 (+629172us) write_op.cc:276] Timestamp: P: 1736542369448960 usec, L: 0
0110 20:52:50.078159 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:50.078468 (+ 309us) log.cc:844] Serialized 354 byte log entry
0110 20:52:50.723294 (+644826us) op_driver.cc:464] REPLICATION: finished
0110 20:52:50.723706 (+ 412us) write_op.cc:301] APPLY: starting
0110 20:52:50.723723 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:50.723759 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:50.723785 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:50.723871 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:50.723908 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:50.723914 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:50.724014 (+ 100us) log.cc:844] Serialized 22 byte log entry
0110 20:52:50.724976 (+ 962us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:50.725015 (+ 39us) write_op.cc:454] Released schema lock
0110 20:52:50.725022 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":348,"num_ops":1,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":778,"prepare.run_wall_time_us":629855,"replication_time_us":645096,"thread_start_us":265,"threads_started":1}]]}
I20250110 20:52:50.726143 3112 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 109) took 1127 ms. Trace:
I20250110 20:52:50.726262 3112 rpcz_store.cc:276] 0110 20:52:49.599052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:49.599147 (+ 95us) service_pool.cc:225] Handling call
0110 20:52:49.599367 (+ 220us) tablet_service.cc:2780] Created scanner 1ab3e0f0087342f29a81b0e18ba6df83 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 78f2d88039e84835a8b8b20ac511998c
0110 20:52:49.600143 (+ 776us) tablet_service.cc:2920] Creating iterator
0110 20:52:49.600176 (+ 33us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:50.078343 (+478167us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:50.724153 (+645810us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1123960 microseconds
0110 20:52:50.724254 (+ 101us) tablet_service.cc:2945] Iterator created
0110 20:52:50.724872 (+ 618us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:50.724898 (+ 26us) tablet_service.cc:3010] has_more: true
0110 20:52:50.724940 (+ 42us) tablet_service.cc:3027] Continuing scan request
0110 20:52:50.725020 (+ 80us) tablet_service.cc:3091] Found scanner 1ab3e0f0087342f29a81b0e18ba6df83 for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 78f2d88039e84835a8b8b20ac511998c
0110 20:52:50.726087 (+ 1067us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":42496}
I20250110 20:52:50.726600 3114 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60872 (request call id 110) took 1127 ms. Trace:
I20250110 20:52:50.726699 3114 rpcz_store.cc:276] 0110 20:52:49.599170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:49.599232 (+ 62us) service_pool.cc:225] Handling call
0110 20:52:49.599390 (+ 158us) tablet_service.cc:2780] Created scanner 5c651b458dce400cb477c74f831ab83b for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 69471afe80d44594bb35ab890e06f328
0110 20:52:49.600172 (+ 782us) tablet_service.cc:2920] Creating iterator
0110 20:52:49.600211 (+ 39us) tablet_service.cc:3297] Waiting safe time to advance
0110 20:52:50.078357 (+478146us) tablet_service.cc:3304] Waiting for operations to commit
0110 20:52:50.724153 (+645796us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1123932 microseconds
0110 20:52:50.724254 (+ 101us) tablet_service.cc:2945] Iterator created
0110 20:52:50.724855 (+ 601us) tablet_service.cc:2967] Iterator init: OK
0110 20:52:50.724891 (+ 36us) tablet_service.cc:3010] has_more: true
0110 20:52:50.724940 (+ 49us) tablet_service.cc:3027] Continuing scan request
0110 20:52:50.725020 (+ 80us) tablet_service.cc:3091] Found scanner 5c651b458dce400cb477c74f831ab83b for tablet 257e9d67bed34c4990d8d399fde6e9e3, query id is 69471afe80d44594bb35ab890e06f328
0110 20:52:50.726545 (+ 1525us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":11776}
W20250110 20:52:50.726943 3246 scanner-internal.cc:458] Time spent opening tablet: real 1.129s user 0.001s sys 0.000s
W20250110 20:52:50.727347 3243 scanner-internal.cc:458] Time spent opening tablet: real 1.129s user 0.001s sys 0.000s
I20250110 20:52:50.830590 2873 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:52:50.831856 3134 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:52:50.840055 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:50.840265 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:50.841018 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:50.841506 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:50.842042 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:50.842958 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:50.843344 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:50.989490 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:50.992651 3004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
0ec47a07211e43cda060a0addfeacf36 | 127.0.104.62:34885 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:44999 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
2ec9211af4dd4e1f9eb696659f945644 | 127.0.104.1:39435 | HEALTHY | <none> | 0 | 0
372d84f3d6134989b052a99839ab6a65 | 127.0.104.3:44505 | HEALTHY | <none> | 1 | 0
730f5dd0bdf9484f805e503c0e4754f7 | 127.0.104.2:44259 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:39435
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:44259
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:44505
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:39435
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:44259
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:44505
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:44999 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 20:52:51.037555 416 log_verifier.cc:126] Checking tablet 257e9d67bed34c4990d8d399fde6e9e3
I20250110 20:52:51.038637 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:51.038857 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.039633 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.039965 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.040381 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.040973 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.041450 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:51.044529 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:51.044719 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.045691 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.046065 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.046353 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.046595 2892 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730f5dd0bdf9484f805e503c0e4754f7 in term 1.
I20250110 20:52:51.047045 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7; no voters: 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.047466 3347 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:52:51.047580 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:51.047659 3347 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:52:51.050494 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.051591 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65"
I20250110 20:52:51.052470 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 election: Requested vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.052865 3022 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.053386 3472 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.054507 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644"
I20250110 20:52:51.055089 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.055392 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250110 20:52:51.059479 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:51.059667 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.060468 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.060962 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.061841 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.062390 3022 raft_consensus.cc:2388] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ec9211af4dd4e1f9eb696659f945644 in current term 2: Already voted for candidate 730f5dd0bdf9484f805e503c0e4754f7 in this term.
I20250110 20:52:51.062584 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.063064 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.063253 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.063947 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.064256 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.064570 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.064656 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.064903 3430 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:52:51.065199 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.066783 3022 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.067184 3430 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:51.067337 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.067656 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.067934 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:51.068426 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.068881 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.069378 3022 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.069856 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.070394 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.071017 3022 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.071020 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.071568 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.072299 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.072741 3022 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.072788 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.073141 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.074591 3022 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.075127 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.075531 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.075538 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.075681 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.075765 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.076231 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.076644 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.077139 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.077404 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.077597 3022 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2ec9211af4dd4e1f9eb696659f945644 in term 2.
I20250110 20:52:51.077822 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.078020 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7; no voters: 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.078286 3430 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:52:51.078380 3430 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:52:51.078478 3430 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:52:51.078328 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.078666 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.079799 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.080336 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.081324 3430 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.081676 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 3. Ops: []
I20250110 20:52:51.082394 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65"
I20250110 20:52:51.082520 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:51.082926 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7"
I20250110 20:52:51.082657 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.083119 3430 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 3 election: Requested vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.083154 3022 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:52:51.083191 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 1.50, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.084792 3472 raft_consensus.cc:3049] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:52:51.084885 3472 raft_consensus.cc:738] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 372d84f3d6134989b052a99839ab6a65, State: Running, Role: LEADER
I20250110 20:52:51.084997 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 3. Ops: []
I20250110 20:52:51.085162 3472 consensus_queue.cc:260] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 50, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.085486 3472 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:52:51.086015 3022 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ec9211af4dd4e1f9eb696659f945644 in term 3.
I20250110 20:52:51.086263 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 3. Ops: []
I20250110 20:52:51.086397 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7; no voters: 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.086822 3430 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 FOLLOWER]: Leader election won for term 3
W20250110 20:52:51.085491 3503 consensus_queue.cc:1159] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "2ec9211af4dd4e1f9eb696659f945644" responder_term: 3 status { last_received { term: 1 index: 50 } last_committed_idx: 50 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 1. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20250110 20:52:51.087604 3430 raft_consensus.cc:695] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 LEADER]: Becoming Leader. State: Replica: 2ec9211af4dd4e1f9eb696659f945644, State: Running, Role: LEADER
I20250110 20:52:51.088128 3430 consensus_queue.cc:237] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 50, Committed index: 50, Last appended: 1.50, Last appended by leader: 50, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.091524 3503 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.091732 3503 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.091987 3024 raft_consensus.cc:1270] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Refusing update from remote peer 2ec9211af4dd4e1f9eb696659f945644: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 3 index: 51. (index mismatch)
I20250110 20:52:51.092397 3154 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:52:51.092235 2750 catalog_manager.cc:5526] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 reported cstate change: term changed from 1 to 3, leader changed from 372d84f3d6134989b052a99839ab6a65 (127.0.104.3) to 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1). New cstate: current_term: 3 leader_uuid: "2ec9211af4dd4e1f9eb696659f945644" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } health_report { overall_health: UNKNOWN } } }
I20250110 20:52:51.093217 3503 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.095060 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.095393 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.095564 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.096086 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.096429 3153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 4 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.096833 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 3 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.097052 3024 raft_consensus.cc:2388] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 372d84f3d6134989b052a99839ab6a65 in current term 3: Already voted for candidate 2ec9211af4dd4e1f9eb696659f945644 in this term.
I20250110 20:52:51.097421 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.098133 416 log_verifier.cc:177] Verified matching terms for 50 ops in tablet 257e9d67bed34c4990d8d399fde6e9e3
I20250110 20:52:51.098091 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 4 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.098546 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.098822 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:51.099783 3154 raft_consensus.cc:1270] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Refusing update from remote peer 2ec9211af4dd4e1f9eb696659f945644: Log matching property violated. Preceding OpId in replica: term: 1 index: 50. Preceding OpId from leader: term: 3 index: 51. (index mismatch)
I20250110 20:52:51.099941 3503 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:51.100531 3430 consensus_queue.cc:1035] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.100925 3504 consensus_queue.cc:1035] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 50, Time since last communication: 0.000s
I20250110 20:52:51.115900 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2ec9211af4dd4e1f9eb696659f945644)
I20250110 20:52:51.116135 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.116811 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.117645 3153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.118531 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.119459 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.119766 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:52:51.121063 3503 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2ec9211af4dd4e1f9eb696659f945644)
I20250110 20:52:51.121223 3503 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.122140 3503 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.122530 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.122588 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.122802 3024 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 372d84f3d6134989b052a99839ab6a65 in term 3.
I20250110 20:52:51.123260 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7; no voters:
I20250110 20:52:51.123724 3503 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250110 20:52:51.123813 3503 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2ec9211af4dd4e1f9eb696659f945644)
I20250110 20:52:51.123898 3503 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:52:51.126734 3503 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.127139 3154 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 FOLLOWER]: Rejecting Update request from peer 2ec9211af4dd4e1f9eb696659f945644 for earlier term 3. Current term is 4. Ops: []
I20250110 20:52:51.127481 3503 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 4 election: Requested vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.128448 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644"
I20250110 20:52:51.128651 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7"
I20250110 20:52:51.128841 3024 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:52:51.129571 3504 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 }, Status: INVALID_TERM, Last received: 3.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250110 20:52:51.131108 3024 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 372d84f3d6134989b052a99839ab6a65 in term 4.
I20250110 20:52:51.131269 3154 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 FOLLOWER]: Rejecting Update request from peer 2ec9211af4dd4e1f9eb696659f945644 for earlier term 3. Current term is 4. Ops: []
I20250110 20:52:51.131489 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644
I20250110 20:52:51.131742 3503 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 FOLLOWER]: Leader election won for term 4
I20250110 20:52:51.131989 3503 raft_consensus.cc:695] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 LEADER]: Becoming Leader. State: Replica: 372d84f3d6134989b052a99839ab6a65, State: Running, Role: LEADER
I20250110 20:52:51.132284 3503 consensus_queue.cc:237] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 3.51, Last appended by leader: 51, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.132645 3504 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 }, Status: INVALID_TERM, Last received: 3.51, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
W20250110 20:52:51.133654 3472 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 4 is higher.
I20250110 20:52:51.135567 2893 raft_consensus.cc:3049] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 LEADER]: Stepping down as leader of term 3
I20250110 20:52:51.135670 2893 raft_consensus.cc:738] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2ec9211af4dd4e1f9eb696659f945644, State: Running, Role: LEADER
I20250110 20:52:51.135991 2893 consensus_queue.cc:260] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 3.51, Last appended by leader: 51, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.136301 2893 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:52:51.135815 2751 catalog_manager.cc:5526] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 reported cstate change: term changed from 3 to 4, leader changed from 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1) to 372d84f3d6134989b052a99839ab6a65 (127.0.104.3). New cstate: current_term: 4 leader_uuid: "372d84f3d6134989b052a99839ab6a65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } health_report { overall_health: UNKNOWN } } }
I20250110 20:52:51.139819 3024 raft_consensus.cc:1270] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Refusing update from remote peer 372d84f3d6134989b052a99839ab6a65: Log matching property violated. Preceding OpId in replica: term: 3 index: 51. Preceding OpId from leader: term: 4 index: 52. (index mismatch)
I20250110 20:52:51.140483 2893 raft_consensus.cc:1270] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 4 FOLLOWER]: Refusing update from remote peer 372d84f3d6134989b052a99839ab6a65: Log matching property violated. Preceding OpId in replica: term: 3 index: 51. Preceding OpId from leader: term: 4 index: 52. (index mismatch)
I20250110 20:52:51.141428 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.141651 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.141295 3503 consensus_queue.cc:1035] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.001s
I20250110 20:52:51.142045 3466 consensus_queue.cc:1035] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
I20250110 20:52:51.142388 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.142800 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.143347 3504 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.143505 3504 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.144270 3504 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.146206 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 5 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.146485 2893 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730f5dd0bdf9484f805e503c0e4754f7 in term 4.
I20250110 20:52:51.147516 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7; no voters: 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.147827 3347 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250110 20:52:51.147929 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 372d84f3d6134989b052a99839ab6a65)
I20250110 20:52:51.148041 3347 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 4 FOLLOWER]: Advancing to term 5
I20250110 20:52:51.150588 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
W20250110 20:52:51.151679 2829 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.3:44505 timed out after 0.007s (SENT) [suppressed 1 similar messages]
I20250110 20:52:51.152273 3153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65"
I20250110 20:52:51.153303 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 5 election: Requested vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.153420 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 5 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644"
I20250110 20:52:51.153513 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 5. Ops: []
I20250110 20:52:51.153999 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 730f5dd0bdf9484f805e503c0e4754f7; no voters: 2ec9211af4dd4e1f9eb696659f945644, 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.154397 3347 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250110 20:52:51.154506 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.154911 3153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 5 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.155275 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 5 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
W20250110 20:52:51.156385 3154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.1:34129 (request call id 54) took 11 ms (client timeout 7 ms). Trace:
I20250110 20:52:51.156497 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 5. Ops: []
W20250110 20:52:51.156517 3154 rpcz_store.cc:269] 0110 20:52:51.144925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:51.150570 (+ 5645us) service_pool.cc:225] Handling call
0110 20:52:51.156314 (+ 5744us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:52:51.157043 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.158723 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 5. Ops: []
I20250110 20:52:51.159307 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
W20250110 20:52:51.161123 3472 raft_consensus.cc:931] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 4 is higher.
W20250110 20:52:51.161412 2829 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505): Timed out: RequestConsensusVote RPC to 127.0.104.3:44505 timed out after 0.007s (SENT)
I20250110 20:52:51.161489 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 5. Ops: []
I20250110 20:52:51.161929 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.162178 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 5 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.162101 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.162791 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.163470 3024 raft_consensus.cc:2388] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2ec9211af4dd4e1f9eb696659f945644 in current term 5: Already voted for candidate 730f5dd0bdf9484f805e503c0e4754f7 in this term.
I20250110 20:52:51.163543 2892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644" is_pre_election: true
I20250110 20:52:51.163774 2892 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 730f5dd0bdf9484f805e503c0e4754f7 in term 4.
I20250110 20:52:51.163868 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.164032 2828 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644; no voters: 372d84f3d6134989b052a99839ab6a65, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.164160 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7; no voters:
I20250110 20:52:51.164321 3517 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 4 FOLLOWER]: Advancing to term 5
I20250110 20:52:51.164415 3347 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250110 20:52:51.164525 3347 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:52:51.164611 3347 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 5 FOLLOWER]: Advancing to term 6
I20250110 20:52:51.165184 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.167007 3517 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 20:52:51.168687 3347 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.169114 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
W20250110 20:52:51.169200 2960 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.3:44505 timed out after 0.006s (SENT) [suppressed 1 similar messages]
I20250110 20:52:51.170029 3519 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.171227 2892 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 5 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 5. Ops: []
I20250110 20:52:51.171423 3153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.171756 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.172499 3347 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 6 election: Requested vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435)
I20250110 20:52:51.173038 3153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65"
W20250110 20:52:51.173218 3154 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.2:37917 (request call id 42) took 9 ms (client timeout 6 ms). Trace:
I20250110 20:52:51.173596 2892 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 5 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 5. Ops: []
I20250110 20:52:51.173681 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.173592 2893 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "730f5dd0bdf9484f805e503c0e4754f7" candidate_term: 6 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "2ec9211af4dd4e1f9eb696659f945644"
I20250110 20:52:51.173864 2893 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 5 FOLLOWER]: Advancing to term 6
W20250110 20:52:51.173318 3154 rpcz_store.cc:269] 0110 20:52:51.163337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:51.163764 (+ 427us) service_pool.cc:225] Handling call
0110 20:52:51.173154 (+ 9390us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:52:51.174839 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 2811
I20250110 20:52:51.175300 3519 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.175521 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.176517 2893 raft_consensus.cc:2463] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 730f5dd0bdf9484f805e503c0e4754f7 in term 6.
I20250110 20:52:51.176927 2958 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7; no voters: 372d84f3d6134989b052a99839ab6a65
I20250110 20:52:51.177276 3347 raft_consensus.cc:2798] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 FOLLOWER]: Leader election won for term 6
I20250110 20:52:51.177891 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.178388 3503 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.179386 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.181383 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.182808 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.183349 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.185374 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.185926 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.187374 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.187738 3517 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:51.187881 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.187883 3517 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.188767 2933 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:52:51.188978 3154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 7 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "372d84f3d6134989b052a99839ab6a65" is_pre_election: true
I20250110 20:52:51.190194 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "2ec9211af4dd4e1f9eb696659f945644" candidate_term: 7 candidate_status { last_received { term: 4 index: 52 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.191951 3347 raft_consensus.cc:695] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 LEADER]: Becoming Leader. State: Replica: 730f5dd0bdf9484f805e503c0e4754f7, State: Running, Role: LEADER
I20250110 20:52:51.192571 3347 consensus_queue.cc:237] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 4.52, Last appended by leader: 52, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.194212 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 LEADER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.195478 3466 consensus_queue.cc:1046] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 }, Status: INVALID_TERM, Last received: 4.52, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.195731 3466 consensus_queue.cc:1225] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [LEADER]: Peer 730f5dd0bdf9484f805e503c0e4754f7 log is divergent from this leader: its last log entry 6.53 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 52
I20250110 20:52:51.196532 3154 raft_consensus.cc:3049] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 LEADER]: Stepping down as leader of term 4
I20250110 20:52:51.196655 3154 raft_consensus.cc:738] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 372d84f3d6134989b052a99839ab6a65, State: Running, Role: LEADER
I20250110 20:52:51.196913 3154 consensus_queue.cc:260] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 4.52, Last appended by leader: 52, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.197211 3154 raft_consensus.cc:3054] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 4 FOLLOWER]: Advancing to term 6
I20250110 20:52:51.197263 3024 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [term 6 LEADER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.197263 2751 catalog_manager.cc:5526] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 reported cstate change: term changed from 4 to 6, leader changed from 372d84f3d6134989b052a99839ab6a65 (127.0.104.3) to 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2). New cstate: current_term: 6 leader_uuid: "730f5dd0bdf9484f805e503c0e4754f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } health_report { overall_health: HEALTHY } } }
I20250110 20:52:51.200065 3154 raft_consensus.cc:1270] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Refusing update from remote peer 730f5dd0bdf9484f805e503c0e4754f7: Log matching property violated. Preceding OpId in replica: term: 4 index: 52. Preceding OpId from leader: term: 6 index: 53. (index mismatch)
I20250110 20:52:51.200996 3347 consensus_queue.cc:1035] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20250110 20:52:51.212340 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.212565 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.213369 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.214929 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
W20250110 20:52:51.221621 3088 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.1:39435 timed out after 0.008s (SENT) [suppressed 2 similar messages]
W20250110 20:52:51.230880 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Timed out: RequestConsensusVote RPC to 127.0.104.1:39435 timed out after 0.008s (SENT)
I20250110 20:52:51.231034 3088 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.231496 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:52:51.364933 3517 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 372d84f3d6134989b052a99839ab6a65 (127.0.104.3:44505), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.365314 2893 raft_consensus.cc:1235] T 257e9d67bed34c4990d8d399fde6e9e3 P 2ec9211af4dd4e1f9eb696659f945644 [term 6 FOLLOWER]: Rejecting Update request from peer 372d84f3d6134989b052a99839ab6a65 for earlier term 4. Current term is 6. Ops: []
I20250110 20:52:51.365669 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 2811
I20250110 20:52:51.393394 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 2942
W20250110 20:52:51.399550 2958 consensus_peers.cc:487] T 257e9d67bed34c4990d8d399fde6e9e3 P 730f5dd0bdf9484f805e503c0e4754f7 -> Peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Couldn't send request to peer 2ec9211af4dd4e1f9eb696659f945644. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:52:51.400038 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.400238 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.400965 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
I20250110 20:52:51.402572 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.407418 3064 generic_service.cc:196] Checking for leaks (request via RPC)
W20250110 20:52:51.416960 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111)
I20250110 20:52:51.417189 3088 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.417498 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:52:51.426153 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.426362 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.427114 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
W20250110 20:52:51.428458 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111)
W20250110 20:52:51.452257 3089 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Timed out: RequestConsensusVote RPC to 127.0.104.2:44259 timed out after 0.012s (SENT)
I20250110 20:52:51.452432 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.452745 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:52:51.459283 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.459475 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.460139 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
W20250110 20:52:51.461428 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111)
W20250110 20:52:51.493350 3089 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Timed out: RequestConsensusVote RPC to 127.0.104.2:44259 timed out after 0.016s (SENT)
I20250110 20:52:51.493512 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.493809 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:52:51.511919 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.512140 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.513139 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
W20250110 20:52:51.515100 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111)
W20250110 20:52:51.548465 3089 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Timed out: RequestConsensusVote RPC to 127.0.104.2:44259 timed out after 0.017s (SENT)
I20250110 20:52:51.548628 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.548952 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:52:51.578107 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.578305 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.578986 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
W20250110 20:52:51.580415 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111)
I20250110 20:52:51.590488 3022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.590492 3023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.591490 3024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.591778 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.592003 3019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.592537 3018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.591401 3020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257e9d67bed34c4990d8d399fde6e9e3" candidate_uuid: "372d84f3d6134989b052a99839ab6a65" candidate_term: 7 candidate_status { last_received { term: 6 index: 53 } } ignore_live_leader: false dest_uuid: "730f5dd0bdf9484f805e503c0e4754f7" is_pre_election: true
I20250110 20:52:51.594161 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 2942
I20250110 20:52:51.594144 3089 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.605873 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:52:51.613744 3087 connection.cc:537] server connection from 127.0.104.2:37917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 20:52:51.613855 3519 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.614032 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 3072
I20250110 20:52:51.614084 3519 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.614825 3519 leader_election.cc:290] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435), 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259)
W20250110 20:52:51.616181 3089 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 730f5dd0bdf9484f805e503c0e4754f7 (127.0.104.2:44259): Network error: Client connection negotiation failed: client connection to 127.0.104.2:44259: connect: Connection refused (error 111)
W20250110 20:52:51.617271 3088 leader_election.cc:336] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 2ec9211af4dd4e1f9eb696659f945644 (127.0.104.1:39435): Network error: Client connection negotiation failed: client connection to 127.0.104.1:39435: connect: Connection refused (error 111)
I20250110 20:52:51.617398 3088 leader_election.cc:304] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 372d84f3d6134989b052a99839ab6a65; no voters: 2ec9211af4dd4e1f9eb696659f945644, 730f5dd0bdf9484f805e503c0e4754f7
I20250110 20:52:51.617656 3519 raft_consensus.cc:2743] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:52:51.620368 3194 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:52:51.811389 3466 raft_consensus.cc:491] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 730f5dd0bdf9484f805e503c0e4754f7)
I20250110 20:52:51.811656 3466 raft_consensus.cc:513] T 257e9d67bed34c4990d8d399fde6e9e3 P 372d84f3d6134989b052a99839ab6a65 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "372d84f3d6134989b052a99839ab6a65" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44505 } } peers { permanent_uuid: "2ec9211af4dd4e1f9eb696659f945644" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39435 } } peers { permanent_uuid: "730f5dd0bdf9484f805e503c0e4754f7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44259 } }
I20250110 20:52:51.812211 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 3072
I20250110 20:52:51.838423 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 2721
I20250110 20:52:51.839437 2782 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:52:51.951846 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 2721
2025-01-10T20:52:51Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (33950 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20250110 20:52:51.991693 416 test_util.cc:274] Using random seed: 1194872921
I20250110 20:52:51.993422 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:52:51.993503 416 ts_itest-base.cc:113] --------------
I20250110 20:52:51.993539 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:52:51.993588 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:52:51.993623 416 ts_itest-base.cc:116] --------------
2025-01-10T20:52:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:52:52Z Disabled control of system clock
I20250110 20:52:52.019954 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33555
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33555 with env {}
W20250110 20:52:52.348575 3541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:52.348909 3541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:52.349030 3541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:52.356946 3541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:52:52.357095 3541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:52.357167 3541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:52:52.357241 3541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:52:52.366798 3541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33555
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:52.368433 3541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:52.370167 3541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:52.380499 3549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:52.380582 3547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:52.380537 3546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:52.381029 3541 server_base.cc:1034] running on GCE node
I20250110 20:52:52.381985 3541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:52.383481 3541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:52.384704 3541 hybrid_clock.cc:648] HybridClock initialized: now 1736542372384635 us; error 43 us; skew 500 ppm
I20250110 20:52:52.385082 3541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:52.387471 3541 webserver.cc:458] Webserver started at http://127.0.104.62:34387/ using document root <none> and password file <none>
I20250110 20:52:52.388247 3541 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:52.388366 3541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:52.388693 3541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:52.391404 3541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "47727b165406405191cd16936164ade5"
format_stamp: "Formatted at 2025-01-10 20:52:52 on dist-test-slave-1h8z"
I20250110 20:52:52.392196 3541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "47727b165406405191cd16936164ade5"
format_stamp: "Formatted at 2025-01-10 20:52:52 on dist-test-slave-1h8z"
I20250110 20:52:52.397646 3541 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 20:52:52.401257 3555 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:52.402685 3541 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250110 20:52:52.402938 3541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "47727b165406405191cd16936164ade5"
format_stamp: "Formatted at 2025-01-10 20:52:52 on dist-test-slave-1h8z"
I20250110 20:52:52.403132 3541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:52.419337 3541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:52.420239 3541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:52.420521 3541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:52.437258 3541 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33555
I20250110 20:52:52.437279 3606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33555 every 8 connection(s)
I20250110 20:52:52.438668 3541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:52:52.439596 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 3541
I20250110 20:52:52.439810 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:52:52.443511 3607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:52.453650 3607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5: Bootstrap starting.
I20250110 20:52:52.456755 3607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:52.457767 3607 log.cc:826] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:52.460409 3607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5: No bootstrap required, opened a new log
I20250110 20:52:52.466794 3607 raft_consensus.cc:357] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47727b165406405191cd16936164ade5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33555 } }
I20250110 20:52:52.467185 3607 raft_consensus.cc:383] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:52.467294 3607 raft_consensus.cc:738] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47727b165406405191cd16936164ade5, State: Initialized, Role: FOLLOWER
I20250110 20:52:52.467922 3607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47727b165406405191cd16936164ade5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33555 } }
I20250110 20:52:52.468124 3607 raft_consensus.cc:397] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:52:52.468247 3607 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:52:52.468397 3607 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:52.470562 3607 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47727b165406405191cd16936164ade5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33555 } }
I20250110 20:52:52.471118 3607 leader_election.cc:304] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47727b165406405191cd16936164ade5; no voters:
I20250110 20:52:52.471690 3607 leader_election.cc:290] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:52:52.471837 3612 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:52:52.472404 3612 raft_consensus.cc:695] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [term 1 LEADER]: Becoming Leader. State: Replica: 47727b165406405191cd16936164ade5, State: Running, Role: LEADER
I20250110 20:52:52.472904 3612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47727b165406405191cd16936164ade5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33555 } }
I20250110 20:52:52.473572 3607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:52:52.475988 3614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47727b165406405191cd16936164ade5. Latest consensus state: current_term: 1 leader_uuid: "47727b165406405191cd16936164ade5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47727b165406405191cd16936164ade5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33555 } } }
I20250110 20:52:52.475968 3613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47727b165406405191cd16936164ade5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47727b165406405191cd16936164ade5" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33555 } } }
I20250110 20:52:52.476372 3614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [sys.catalog]: This master's current role is: LEADER
I20250110 20:52:52.476382 3613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5 [sys.catalog]: This master's current role is: LEADER
I20250110 20:52:52.477748 3619 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:52:52.483619 3619 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:52:52.493001 3619 catalog_manager.cc:1348] Generated new cluster ID: 27715a2d0f5b4af2a76365ef24d16714
I20250110 20:52:52.493188 3619 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:52:52.530686 3619 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:52:52.531972 3619 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:52:52.549616 3619 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 47727b165406405191cd16936164ade5: Generated new TSK 0
I20250110 20:52:52.550518 3619 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:52:52.561378 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33555
--builtin_ntp_servers=127.0.104.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20250110 20:52:52.893959 3631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:52.894251 3631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:52.894344 3631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:52.902207 3631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:52.902419 3631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:52:52.911846 3631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:33555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:52.913367 3631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:52.915150 3631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:52.927938 3637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:52.928002 3636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:52.928575 3639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:52.929391 3631 server_base.cc:1034] running on GCE node
I20250110 20:52:52.930050 3631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:52.931059 3631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:52.932276 3631 hybrid_clock.cc:648] HybridClock initialized: now 1736542372932213 us; error 44 us; skew 500 ppm
I20250110 20:52:52.932575 3631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:52.934837 3631 webserver.cc:458] Webserver started at http://127.0.104.1:46805/ using document root <none> and password file <none>
I20250110 20:52:52.935587 3631 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:52.935704 3631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:52.936018 3631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:52.938652 3631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b286bd11c62645c1a74096177d504b31"
format_stamp: "Formatted at 2025-01-10 20:52:52 on dist-test-slave-1h8z"
I20250110 20:52:52.939466 3631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b286bd11c62645c1a74096177d504b31"
format_stamp: "Formatted at 2025-01-10 20:52:52 on dist-test-slave-1h8z"
I20250110 20:52:52.944707 3631 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20250110 20:52:52.948155 3645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:52.949678 3631 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250110 20:52:52.949887 3631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b286bd11c62645c1a74096177d504b31"
format_stamp: "Formatted at 2025-01-10 20:52:52 on dist-test-slave-1h8z"
I20250110 20:52:52.950064 3631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:52.975793 3631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:52.976621 3631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:52.976897 3631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:52.977983 3631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:52:52.979598 3631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:52:52.979673 3631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:52.979763 3631 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:52:52.979812 3631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.009194 3631 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35003
I20250110 20:52:53.009246 3757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35003 every 8 connection(s)
I20250110 20:52:53.010512 3631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:52:53.011207 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 3631
I20250110 20:52:53.011413 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:52:53.015117 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33555
--builtin_ntp_servers=127.0.104.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250110 20:52:53.024683 3758 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33555
I20250110 20:52:53.025091 3758 heartbeater.cc:463] Registering TS with master...
I20250110 20:52:53.026167 3758 heartbeater.cc:510] Master 127.0.104.62:33555 requested a full tablet report, sending...
I20250110 20:52:53.028533 3572 ts_manager.cc:194] Registered new tserver with Master: b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003)
I20250110 20:52:53.030128 3572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:59705
W20250110 20:52:53.347066 3762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:53.347348 3762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:53.347486 3762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:53.355286 3762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:53.355526 3762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:52:53.364871 3762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:33555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:53.366487 3762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:53.368294 3762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:53.379671 3767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:53.379740 3768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:53.379704 3770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:53.381032 3762 server_base.cc:1034] running on GCE node
I20250110 20:52:53.381673 3762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:53.382656 3762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:53.383909 3762 hybrid_clock.cc:648] HybridClock initialized: now 1736542373383840 us; error 47 us; skew 500 ppm
I20250110 20:52:53.384223 3762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:53.386570 3762 webserver.cc:458] Webserver started at http://127.0.104.2:36185/ using document root <none> and password file <none>
I20250110 20:52:53.387390 3762 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:53.387516 3762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:53.387838 3762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:53.391981 3762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c8049f77e2ff44129e129bab33c87b9b"
format_stamp: "Formatted at 2025-01-10 20:52:53 on dist-test-slave-1h8z"
I20250110 20:52:53.392800 3762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c8049f77e2ff44129e129bab33c87b9b"
format_stamp: "Formatted at 2025-01-10 20:52:53 on dist-test-slave-1h8z"
I20250110 20:52:53.398072 3762 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 20:52:53.401886 3776 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.403432 3762 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:52:53.403632 3762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c8049f77e2ff44129e129bab33c87b9b"
format_stamp: "Formatted at 2025-01-10 20:52:53 on dist-test-slave-1h8z"
I20250110 20:52:53.403856 3762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:53.421285 3762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:53.422160 3762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:53.422432 3762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:53.423417 3762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:52:53.425050 3762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:52:53.425132 3762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.425256 3762 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:52:53.425307 3762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.459481 3762 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38847
I20250110 20:52:53.459512 3888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38847 every 8 connection(s)
I20250110 20:52:53.460827 3762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:52:53.465930 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 3762
I20250110 20:52:53.466120 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:52:53.468937 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33555
--builtin_ntp_servers=127.0.104.20:39763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250110 20:52:53.473721 3889 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33555
I20250110 20:52:53.474108 3889 heartbeater.cc:463] Registering TS with master...
I20250110 20:52:53.475270 3889 heartbeater.cc:510] Master 127.0.104.62:33555 requested a full tablet report, sending...
I20250110 20:52:53.477042 3572 ts_manager.cc:194] Registered new tserver with Master: c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:52:53.477833 3572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:48221
W20250110 20:52:53.801105 3892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:52:53.801410 3892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:52:53.801533 3892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:52:53.809404 3892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:52:53.809645 3892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:52:53.819169 3892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.0.104.62:33555
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:52:53.820784 3892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:52:53.822551 3892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:52:53.834317 3897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:53.834347 3898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:52:53.834436 3900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:52:53.835001 3892 server_base.cc:1034] running on GCE node
I20250110 20:52:53.835606 3892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:52:53.836549 3892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:52:53.837769 3892 hybrid_clock.cc:648] HybridClock initialized: now 1736542373837695 us; error 46 us; skew 500 ppm
I20250110 20:52:53.838091 3892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:52:53.840380 3892 webserver.cc:458] Webserver started at http://127.0.104.3:41137/ using document root <none> and password file <none>
I20250110 20:52:53.841128 3892 fs_manager.cc:362] Metadata directory not provided
I20250110 20:52:53.841248 3892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:52:53.841558 3892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:52:53.844234 3892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
format_stamp: "Formatted at 2025-01-10 20:52:53 on dist-test-slave-1h8z"
I20250110 20:52:53.844981 3892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
format_stamp: "Formatted at 2025-01-10 20:52:53 on dist-test-slave-1h8z"
I20250110 20:52:53.850232 3892 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250110 20:52:53.853802 3906 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.855319 3892 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:52:53.855520 3892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
format_stamp: "Formatted at 2025-01-10 20:52:53 on dist-test-slave-1h8z"
I20250110 20:52:53.855702 3892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:52:53.887241 3892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:52:53.888098 3892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:52:53.888366 3892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:52:53.889305 3892 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:52:53.890856 3892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:52:53.890936 3892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.891057 3892 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:52:53.891112 3892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:52:53.920358 3892 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:37837
I20250110 20:52:53.920401 4018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:37837 every 8 connection(s)
I20250110 20:52:53.921634 3892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:52:53.929718 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 3892
I20250110 20:52:53.929862 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:52:53.934386 4019 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33555
I20250110 20:52:53.934679 4019 heartbeater.cc:463] Registering TS with master...
I20250110 20:52:53.935504 4019 heartbeater.cc:510] Master 127.0.104.62:33555 requested a full tablet report, sending...
I20250110 20:52:53.936820 3572 ts_manager.cc:194] Registered new tserver with Master: ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:53.937589 3572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:53959
I20250110 20:52:53.945683 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:52:53.948942 416 test_util.cc:274] Using random seed: 1196830196
I20250110 20:52:53.966866 3572 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52264:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:52:53.968379 3572 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:52:53.993861 3693 tablet_service.cc:1467] Processing CreateTablet for tablet 26db1d18dd4541db812c55f643f305b3 (DEFAULT_TABLE table=test-workload [id=81e8d00cf3cf461297f602ba74ce5cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:52:53.993911 3824 tablet_service.cc:1467] Processing CreateTablet for tablet 26db1d18dd4541db812c55f643f305b3 (DEFAULT_TABLE table=test-workload [id=81e8d00cf3cf461297f602ba74ce5cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:52:53.995330 3693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26db1d18dd4541db812c55f643f305b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:53.995435 3824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26db1d18dd4541db812c55f643f305b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:53.998260 3954 tablet_service.cc:1467] Processing CreateTablet for tablet 26db1d18dd4541db812c55f643f305b3 (DEFAULT_TABLE table=test-workload [id=81e8d00cf3cf461297f602ba74ce5cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:52:53.999873 3954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26db1d18dd4541db812c55f643f305b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:52:54.006028 4037 tablet_bootstrap.cc:492] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31: Bootstrap starting.
I20250110 20:52:54.009843 4037 tablet_bootstrap.cc:654] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:54.012743 4037 log.cc:826] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:54.021914 4040 tablet_bootstrap.cc:492] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2: Bootstrap starting.
I20250110 20:52:54.023368 4038 tablet_bootstrap.cc:492] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Bootstrap starting.
I20250110 20:52:54.024549 4037 tablet_bootstrap.cc:492] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31: No bootstrap required, opened a new log
I20250110 20:52:54.024863 4037 ts_tablet_manager.cc:1397] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20250110 20:52:54.027518 4040 tablet_bootstrap.cc:654] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:54.027518 4038 tablet_bootstrap.cc:654] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Neither blocks nor log segments found. Creating new log.
I20250110 20:52:54.028584 4040 log.cc:826] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:54.028716 4038 log.cc:826] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Log is configured to *not* fsync() on all Append() calls
I20250110 20:52:54.031677 4038 tablet_bootstrap.cc:492] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: No bootstrap required, opened a new log
I20250110 20:52:54.031684 4040 tablet_bootstrap.cc:492] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2: No bootstrap required, opened a new log
I20250110 20:52:54.031972 4038 ts_tablet_manager.cc:1397] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20250110 20:52:54.031976 4040 ts_tablet_manager.cc:1397] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20250110 20:52:54.033910 3758 heartbeater.cc:502] Master 127.0.104.62:33555 was elected leader, sending a full tablet report...
I20250110 20:52:54.035192 4037 raft_consensus.cc:357] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.035647 4037 raft_consensus.cc:383] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:54.035758 4037 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b286bd11c62645c1a74096177d504b31, State: Initialized, Role: FOLLOWER
I20250110 20:52:54.036690 4037 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.037184 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:54.037439 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.039589 4038 raft_consensus.cc:357] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.040071 4038 raft_consensus.cc:383] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:54.040195 4038 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8049f77e2ff44129e129bab33c87b9b, State: Initialized, Role: FOLLOWER
I20250110 20:52:54.041016 4038 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.042446 4047 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:54.042515 3889 heartbeater.cc:502] Master 127.0.104.62:33555 was elected leader, sending a full tablet report...
I20250110 20:52:54.042806 4047 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.043852 4040 raft_consensus.cc:357] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.044296 4040 raft_consensus.cc:383] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:52:54.044409 4040 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca06c6ed33684aafbaebc0ddd43546f2, State: Initialized, Role: FOLLOWER
I20250110 20:52:54.048134 4037 ts_tablet_manager.cc:1428] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31: Time spent starting tablet: real 0.023s user 0.007s sys 0.004s
W20250110 20:52:54.051788 3777 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.1:35003 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20250110 20:52:54.052728 4040 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.053210 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.053285 4048 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:54.053541 4048 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.067893 4038 ts_tablet_manager.cc:1428] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Time spent starting tablet: real 0.036s user 0.019s sys 0.011s
I20250110 20:52:54.068471 4048 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:52:54.069664 4019 heartbeater.cc:502] Master 127.0.104.62:33555 was elected leader, sending a full tablet report...
W20250110 20:52:54.069744 3648 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.2:38847 for RPC RequestConsensusVote timed out after 0.008s (ON_OUTBOUND_QUEUE)
W20250110 20:52:54.075356 3907 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.1:35003 for RPC RequestConsensusVote timed out after 0.006s (ON_OUTBOUND_QUEUE)
I20250110 20:52:54.076817 4047 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
W20250110 20:52:54.084451 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.006s (SENT)
I20250110 20:52:54.084784 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.085270 3713 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca06c6ed33684aafbaebc0ddd43546f2 in term 0.
I20250110 20:52:54.088084 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.088361 3713 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 0.
I20250110 20:52:54.088867 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b; no voters:
I20250110 20:52:54.089309 4047 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:52:54.089448 4047 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:52:54.089535 4047 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:54.090144 4040 ts_tablet_manager.cc:1428] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2: Time spent starting tablet: real 0.058s user 0.014s sys 0.010s
I20250110 20:52:54.092551 4047 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.093180 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.093252 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.093400 4047 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 1 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.093634 3974 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 0.
I20250110 20:52:54.093686 3844 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b286bd11c62645c1a74096177d504b31 in current term 1: Already voted for candidate c8049f77e2ff44129e129bab33c87b9b in this term.
W20250110 20:52:54.093806 3779 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837): Timed out: RequestConsensusVote RPC to 127.0.104.3:37837 timed out after 0.006s (SENT)
I20250110 20:52:54.094345 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:52:54.094528 3974 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:54.094691 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:52:54.094911 3713 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:52:54.095151 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.095090 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.095346 3844 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 in current term 1: Already voted for candidate c8049f77e2ff44129e129bab33c87b9b in this term.
I20250110 20:52:54.096156 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:52:54.097878 3974 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 1.
I20250110 20:52:54.097882 3713 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 1.
I20250110 20:52:54.098063 4051 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:52:54.098495 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:52:54.098839 4047 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:52:54.099006 4047 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 LEADER]: Becoming Leader. State: Replica: c8049f77e2ff44129e129bab33c87b9b, State: Running, Role: LEADER
I20250110 20:52:54.098965 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:54.099453 4047 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.099606 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:52:54.105441 4051 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:54.105655 4051 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.106438 4051 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:52:54.107198 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.107410 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.104159 3572 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b reported cstate change: term changed from 0 to 1, leader changed from <none> to c8049f77e2ff44129e129bab33c87b9b (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c8049f77e2ff44129e129bab33c87b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20250110 20:52:54.108345 3907 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:52:54.109004 4051 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:54.111915 4047 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:52:54.118556 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:52:54.118774 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.120196 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.120813 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.121624 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.122176 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:54.133342 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:54.133800 4051 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:52:54.133970 4051 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.134706 4051 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:52:54.135151 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.135380 3713 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca06c6ed33684aafbaebc0ddd43546f2 in term 1.
I20250110 20:52:54.135730 3907 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:52:54.136020 4051 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:52:54.136111 4051 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:52:54.136216 4051 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:52:54.137145 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.138927 4051 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.139262 3974 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 FOLLOWER]: Rejecting Update request from peer c8049f77e2ff44129e129bab33c87b9b for earlier term 1. Current term is 2. Ops: []
I20250110 20:52:54.140338 4046 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:52:54.140658 4046 raft_consensus.cc:3049] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:52:54.140771 4046 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8049f77e2ff44129e129bab33c87b9b, State: Running, Role: LEADER
I20250110 20:52:54.141072 4046 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.141678 4046 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:52:54.143380 4051 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 2 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:52:54.143760 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:52:54.143960 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:52:54.146272 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:52:54.146486 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.146864 3844 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca06c6ed33684aafbaebc0ddd43546f2 in term 2.
I20250110 20:52:54.147395 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31
I20250110 20:52:54.147672 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.147737 4051 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:52:54.148001 4051 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 LEADER]: Becoming Leader. State: Replica: ca06c6ed33684aafbaebc0ddd43546f2, State: Running, Role: LEADER
I20250110 20:52:54.148499 4051 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.148640 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.148730 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.148886 3844 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b286bd11c62645c1a74096177d504b31 in current term 2: Already voted for candidate ca06c6ed33684aafbaebc0ddd43546f2 in this term.
I20250110 20:52:54.149392 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:54.149698 4043 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:52:54.152159 3844 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Refusing update from remote peer ca06c6ed33684aafbaebc0ddd43546f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250110 20:52:54.152312 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:52:54.153033 3713 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Refusing update from remote peer ca06c6ed33684aafbaebc0ddd43546f2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250110 20:52:54.153091 4051 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:52:54.154098 4046 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:52:54.154268 4046 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.155051 4046 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.155298 4048 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:52:54.155532 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.157151 3572 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 reported cstate change: term changed from 1 to 2, leader changed from c8049f77e2ff44129e129bab33c87b9b (127.0.104.2) to ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3). New cstate: current_term: 2 leader_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: HEALTHY } } }
I20250110 20:52:54.160243 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.161043 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:54.161386 4046 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250110 20:52:54.168581 3803 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37008: Illegal state: replica c8049f77e2ff44129e129bab33c87b9b is not leader of this config: current role FOLLOWER
I20250110 20:52:54.169040 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:54.169245 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.169970 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
W20250110 20:52:54.170641 3804 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:37008: Illegal state: replica c8049f77e2ff44129e129bab33c87b9b is not leader of this config: current role FOLLOWER
I20250110 20:52:54.172508 4046 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:54.172686 4046 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.173374 4046 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.175099 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.172515 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.175838 3844 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 2.
I20250110 20:52:54.175971 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.176535 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.177007 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:54.177438 4046 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:54.179008 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:54.179106 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.179910 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:54.180246 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250110 20:52:54.188944 3672 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47008: Illegal state: replica b286bd11c62645c1a74096177d504b31 is not leader of this config: current role FOLLOWER
W20250110 20:52:54.188923 3673 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:47008: Illegal state: replica b286bd11c62645c1a74096177d504b31 is not leader of this config: current role FOLLOWER
W20250110 20:52:54.207267 4063 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:52:54.207379 4063 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250110 20:52:54.209901 3713 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:52:54.210024 3713 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250110 20:52:54.210214 3843 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:52:54.210292 3843 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250110 20:52:54.972587 4046 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:54.972803 4046 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:54.973830 4046 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:54.974290 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:54.974992 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:54.975539 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:55.156272 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:55.156517 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:55.157267 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:55.157750 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:55.157804 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:55.158438 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:55.158701 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:55.433198 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:55.433435 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:55.434177 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:55.434731 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:55.434787 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:55.434978 4076 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=0, attempt_no=2}) took 1234 ms. Trace:
I20250110 20:52:55.435501 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:55.435140 4076 rpcz_store.cc:276] 0110 20:52:54.200971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:54.201174 (+ 203us) service_pool.cc:225] Handling call
0110 20:52:55.434908 (+1233734us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:54.208214 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:54.208328 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:54.208335 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:54.208337 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:54.208424 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:54.208451 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:54.208456 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:54.208505 (+ 49us) tablet.cc:698] Row locks acquired
0110 20:52:54.208507 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:54.208545 (+ 38us) write_op.cc:270] Start()
0110 20:52:54.225717 (+ 17172us) write_op.cc:276] Timestamp: P: 1736542374208478 usec, L: 0
0110 20:52:54.225722 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:52:54.225966 (+ 244us) log.cc:844] Serialized 345 byte log entry
0110 20:52:55.434180 (+1208214us) op_driver.cc:464] REPLICATION: finished
0110 20:52:55.434313 (+ 133us) write_op.cc:301] APPLY: starting
0110 20:52:55.434332 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:55.434380 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:55.434419 (+ 39us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:55.434519 (+ 100us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:55.434558 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:55.434566 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:55.434657 (+ 91us) log.cc:844] Serialized 22 byte log entry
0110 20:52:55.434778 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:55.434799 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:55.434807 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":1934464,"child_traces":[["op",{"apply.queue_time_us":69,"mutex_wait_us":26,"num_ops":1,"prepare.queue_time_us":3330,"prepare.run_cpu_time_us":769,"prepare.run_wall_time_us":17831,"replication_time_us":1208423}]]}
I20250110 20:52:55.965561 4046 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:56.217531 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:56.219718 4086 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=2, attempt_no=0}) took 1241 ms. Trace:
I20250110 20:52:56.219846 4086 rpcz_store.cc:276] 0110 20:52:54.978585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:54.978648 (+ 63us) service_pool.cc:225] Handling call
0110 20:52:56.219652 (+1241004us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:54.979609 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:54.979756 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:54.979760 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:52:54.979762 (+ 2us) tablet.cc:659] Decoding operations
0110 20:52:54.979849 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:54.979858 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:54.979861 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:54.979903 (+ 42us) tablet.cc:698] Row locks acquired
0110 20:52:54.979905 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:54.979939 (+ 34us) write_op.cc:270] Start()
0110 20:52:55.396100 (+416161us) write_op.cc:276] Timestamp: P: 1736542374979916 usec, L: 0
0110 20:52:55.396106 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:55.396348 (+ 242us) log.cc:844] Serialized 354 byte log entry
0110 20:52:56.218708 (+822360us) op_driver.cc:464] REPLICATION: finished
0110 20:52:56.219171 (+ 463us) write_op.cc:301] APPLY: starting
0110 20:52:56.219190 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:56.219236 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:56.219267 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:56.219351 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:56.219379 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:56.219384 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:56.219464 (+ 80us) log.cc:844] Serialized 22 byte log entry
0110 20:52:56.219549 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:56.219566 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:56.219572 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":394,"num_ops":1,"prepare.queue_time_us":433,"prepare.run_cpu_time_us":763,"prepare.run_wall_time_us":416826,"replication_time_us":822557,"thread_start_us":565,"threads_started":2}]]}
I20250110 20:52:56.560946 4046 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:56.561178 4046 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:56.561940 4046 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:56.562464 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:56.562714 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:56.563309 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:56.564314 4046 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:56.565724 4086 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=3, attempt_no=0}) took 1128 ms. Trace:
I20250110 20:52:56.565824 4086 rpcz_store.cc:276] 0110 20:52:55.437692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:55.437789 (+ 97us) service_pool.cc:225] Handling call
0110 20:52:56.565668 (+1127879us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:55.438356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:55.438476 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:55.438482 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:52:55.438485 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:55.438573 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:55.438584 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:55.438589 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:55.438636 (+ 47us) tablet.cc:698] Row locks acquired
0110 20:52:55.438639 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:52:55.438676 (+ 37us) write_op.cc:270] Start()
0110 20:52:56.111853 (+673177us) write_op.cc:276] Timestamp: P: 1736542375438648 usec, L: 0
0110 20:52:56.111859 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:56.112131 (+ 272us) log.cc:844] Serialized 354 byte log entry
0110 20:52:56.565116 (+452985us) op_driver.cc:464] REPLICATION: finished
0110 20:52:56.565241 (+ 125us) write_op.cc:301] APPLY: starting
0110 20:52:56.565258 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:56.565299 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:56.565325 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:56.565396 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:56.565421 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:56.565427 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:52:56.565490 (+ 63us) log.cc:844] Serialized 22 byte log entry
0110 20:52:56.565560 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:56.565577 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:56.565583 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":777,"prepare.run_wall_time_us":673853,"replication_time_us":453225}]]}
I20250110 20:52:57.049582 4046 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:57.049873 4046 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:57.050673 4046 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:57.051198 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:57.051839 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 7 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:57.052461 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:57.158143 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:57.158366 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:57.159117 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:57.159576 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:57.159662 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 6 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:57.160257 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:57.603231 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:57.605597 4093 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=5, attempt_no=0}) took 1037 ms. Trace:
I20250110 20:52:57.605799 4093 rpcz_store.cc:276] 0110 20:52:56.567637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:56.567684 (+ 47us) service_pool.cc:225] Handling call
0110 20:52:57.605534 (+1037850us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:56.677979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:56.678127 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:56.678134 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:56.678137 (+ 3us) tablet.cc:659] Decoding operations
0110 20:52:56.678234 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:56.678245 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:56.678250 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:56.678305 (+ 55us) tablet.cc:698] Row locks acquired
0110 20:52:56.678307 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:56.678351 (+ 44us) write_op.cc:270] Start()
0110 20:52:56.818514 (+140163us) write_op.cc:276] Timestamp: P: 1736542376678301 usec, L: 0
0110 20:52:56.818520 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:56.818779 (+ 259us) log.cc:844] Serialized 354 byte log entry
0110 20:52:57.604706 (+785927us) op_driver.cc:464] REPLICATION: finished
0110 20:52:57.605098 (+ 392us) write_op.cc:301] APPLY: starting
0110 20:52:57.605113 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:57.605147 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:57.605173 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:57.605241 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:57.605266 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:57.605271 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:52:57.605344 (+ 73us) log.cc:844] Serialized 22 byte log entry
0110 20:52:57.605418 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:57.605435 (+ 17us) write_op.cc:454] Released schema lock
0110 20:52:57.605442 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":334,"num_ops":1,"prepare.queue_time_us":109832,"prepare.run_cpu_time_us":832,"prepare.run_wall_time_us":140890,"replication_time_us":786152,"thread_start_us":243,"threads_started":1}]]}
I20250110 20:52:57.719987 4046 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:58.318634 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:58.318899 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:58.319650 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:58.320510 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:58.321142 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:58.322131 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:58.322453 4100 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=6, attempt_no=0}) took 1267 ms. Trace:
I20250110 20:52:58.322562 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:58.322597 4100 rpcz_store.cc:276] 0110 20:52:57.054774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:57.054835 (+ 61us) service_pool.cc:225] Handling call
0110 20:52:58.322388 (+1267553us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:57.055352 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:57.055448 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:57.055453 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:52:57.055475 (+ 22us) tablet.cc:659] Decoding operations
0110 20:52:57.055570 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:57.055581 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:57.055587 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:57.055629 (+ 42us) tablet.cc:698] Row locks acquired
0110 20:52:57.055630 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:57.055663 (+ 33us) write_op.cc:270] Start()
0110 20:52:57.697802 (+642139us) write_op.cc:276] Timestamp: P: 1736542377055632 usec, L: 0
0110 20:52:57.697808 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:57.698030 (+ 222us) log.cc:844] Serialized 354 byte log entry
0110 20:52:58.320484 (+622454us) op_driver.cc:464] REPLICATION: finished
0110 20:52:58.321702 (+ 1218us) write_op.cc:301] APPLY: starting
0110 20:52:58.321723 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:58.321777 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:58.321846 (+ 69us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:58.321951 (+ 105us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:58.321990 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:58.321998 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:58.322136 (+ 138us) log.cc:844] Serialized 22 byte log entry
0110 20:52:58.322239 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:58.322262 (+ 23us) write_op.cc:454] Released schema lock
0110 20:52:58.322271 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1127,"num_ops":1,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":642748,"replication_time_us":622640,"thread_start_us":1037,"threads_started":1}]]}
I20250110 20:52:58.343415 4046 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:58.343657 4046 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:58.344393 4046 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:58.344918 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:58.344926 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:58.345656 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:58.346567 4046 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:58.704144 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:58.704378 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:58.705149 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:58.705561 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:58.705655 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:58.706243 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:58.706512 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:59.457039 4107 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:59.457288 4107 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:59.458379 4107 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:59.458961 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:52:59.459215 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:59.460753 4110 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=8, attempt_no=0}) took 1136 ms. Trace:
I20250110 20:52:59.460942 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:59.461346 4107 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:52:59.461388 4110 rpcz_store.cc:276] 0110 20:52:58.324642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:58.324710 (+ 68us) service_pool.cc:225] Handling call
0110 20:52:59.460700 (+1135990us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:58.325494 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:58.325590 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:58.325594 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:52:58.325595 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:58.325670 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:58.325679 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:58.325682 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:58.325755 (+ 73us) tablet.cc:698] Row locks acquired
0110 20:52:58.325757 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:52:58.325799 (+ 42us) write_op.cc:270] Start()
0110 20:52:59.035958 (+710159us) write_op.cc:276] Timestamp: P: 1736542378325733 usec, L: 0
0110 20:52:59.035964 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:52:59.036183 (+ 219us) log.cc:844] Serialized 354 byte log entry
0110 20:52:59.458431 (+422248us) op_driver.cc:464] REPLICATION: finished
0110 20:52:59.459605 (+ 1174us) write_op.cc:301] APPLY: starting
0110 20:52:59.459628 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0110 20:52:59.459682 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 20:52:59.459719 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:52:59.459837 (+ 118us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:52:59.459877 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:52:59.459885 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:52:59.460029 (+ 144us) log.cc:844] Serialized 22 byte log entry
0110 20:52:59.460420 (+ 391us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:52:59.460441 (+ 21us) write_op.cc:454] Released schema lock
0110 20:52:59.460450 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":858,"mutex_wait_us":2,"num_ops":1,"prepare.queue_time_us":305,"prepare.run_cpu_time_us":5613,"prepare.run_wall_time_us":715668,"raft.queue_time_us":594,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":422425,"thread_start_us":8111,"threads_started":3}]]}
I20250110 20:52:59.852881 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:52:59.853092 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:52:59.853852 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:52:59.854259 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:52:59.854807 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:52:59.855352 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:52:59.855762 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:00.022256 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:00.022495 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:00.023129 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:00.023643 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:00.023813 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:00.024330 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:00.024909 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:00.025099 4116 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:00.025282 4116 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:00.026013 4116 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:00.026459 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:00.026674 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:00.027272 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:00.027738 4116 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:00.231468 4116 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:00.231688 4116 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:00.232460 4116 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:00.233011 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:00.233079 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:00.233678 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:00.234187 4116 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:00.234424 4118 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=9, attempt_no=0}) took 1887 ms. Trace:
I20250110 20:53:00.234542 4118 rpcz_store.cc:276] 0110 20:52:58.347451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:58.347508 (+ 57us) service_pool.cc:225] Handling call
0110 20:53:00.234375 (+1886867us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:52:59.041213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:52:59.041326 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:52:59.041333 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:52:59.041334 (+ 1us) tablet.cc:659] Decoding operations
0110 20:52:59.041411 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:52:59.041421 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:52:59.041424 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:52:59.041466 (+ 42us) tablet.cc:698] Row locks acquired
0110 20:52:59.041467 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:52:59.041510 (+ 43us) write_op.cc:270] Start()
0110 20:53:00.030651 (+989141us) write_op.cc:276] Timestamp: P: 1736542379041482 usec, L: 0
0110 20:53:00.030655 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:53:00.030897 (+ 242us) log.cc:844] Serialized 354 byte log entry
0110 20:53:00.232364 (+201467us) op_driver.cc:464] REPLICATION: finished
0110 20:53:00.233482 (+ 1118us) write_op.cc:301] APPLY: starting
0110 20:53:00.233504 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:00.233558 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:00.233588 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:00.233706 (+ 118us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:00.233741 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:00.233749 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:00.233898 (+ 149us) log.cc:844] Serialized 43 byte log entry
0110 20:53:00.234179 (+ 281us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:00.234199 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:00.234207 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":997,"num_ops":1,"prepare.queue_time_us":693252,"prepare.run_cpu_time_us":734,"prepare.run_wall_time_us":989799,"raft.queue_time_us":29,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":257,"replication_time_us":201670,"spinlock_wait_cycles":15104,"thread_start_us":463,"threads_started":1}]]}
I20250110 20:53:00.423276 4043 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:00.423544 4043 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:00.424274 4043 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:00.424820 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:00.424846 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:00.425467 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:01.343359 4116 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:01.343549 4116 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:01.344285 4116 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:01.344759 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:01.344903 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:01.345515 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:01.347062 4126 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=10, attempt_no=0}) took 1883 ms. Trace:
I20250110 20:53:01.347175 4126 rpcz_store.cc:276] 0110 20:52:59.463566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:52:59.463624 (+ 58us) service_pool.cc:225] Handling call
0110 20:53:01.346996 (+1883372us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:00.031039 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:00.031159 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:00.031165 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:53:00.031167 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:00.031244 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:00.031254 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:00.031257 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:00.031300 (+ 43us) tablet.cc:698] Row locks acquired
0110 20:53:00.031302 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:00.031342 (+ 40us) write_op.cc:270] Start()
0110 20:53:00.361480 (+330138us) write_op.cc:276] Timestamp: P: 1736542380031318 usec, L: 0
0110 20:53:00.361484 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:53:00.361707 (+ 223us) log.cc:844] Serialized 354 byte log entry
0110 20:53:01.345164 (+983457us) op_driver.cc:464] REPLICATION: finished
0110 20:53:01.346485 (+ 1321us) write_op.cc:301] APPLY: starting
0110 20:53:01.346504 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:01.346567 (+ 63us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:01.346594 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:01.346665 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:01.346694 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:01.346701 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:01.346810 (+ 109us) log.cc:844] Serialized 22 byte log entry
0110 20:53:01.346886 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:01.346903 (+ 17us) write_op.cc:454] Released schema lock
0110 20:53:01.346910 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1242,"num_ops":1,"prepare.queue_time_us":566913,"prepare.run_cpu_time_us":670,"prepare.run_wall_time_us":330733,"replication_time_us":983645,"thread_start_us":514,"threads_started":1}]]}
I20250110 20:53:01.371609 4043 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:01.373262 4126 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=11, attempt_no=0}) took 1136 ms. Trace:
I20250110 20:53:01.373381 4126 rpcz_store.cc:276] 0110 20:53:00.236646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:00.236709 (+ 63us) service_pool.cc:225] Handling call
0110 20:53:01.373197 (+1136488us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:00.361816 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:00.361944 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:00.361951 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:00.361953 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:00.362061 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:00.362074 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:00.362078 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:00.362141 (+ 63us) tablet.cc:698] Row locks acquired
0110 20:53:00.362143 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:00.362192 (+ 49us) write_op.cc:270] Start()
0110 20:53:00.377338 (+ 15146us) write_op.cc:276] Timestamp: P: 1736542380362133 usec, L: 0
0110 20:53:00.377343 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:00.377601 (+ 258us) log.cc:844] Serialized 354 byte log entry
0110 20:53:01.372555 (+994954us) op_driver.cc:464] REPLICATION: finished
0110 20:53:01.372685 (+ 130us) write_op.cc:301] APPLY: starting
0110 20:53:01.372703 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:01.372749 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:01.372780 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:01.372866 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:01.372903 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:01.372910 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:01.372992 (+ 82us) log.cc:844] Serialized 22 byte log entry
0110 20:53:01.373072 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:01.373092 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:01.373100 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"num_ops":1,"prepare.queue_time_us":124637,"prepare.run_cpu_time_us":806,"prepare.run_wall_time_us":15866,"replication_time_us":995183}]]}
I20250110 20:53:01.994503 4116 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:02.385488 4116 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:02.385691 4116 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:02.386462 4116 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:02.386981 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:02.386986 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:02.387979 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:02.388319 4136 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=12, attempt_no=0}) took 1039 ms. Trace:
I20250110 20:53:02.388375 4116 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:02.388428 4136 rpcz_store.cc:276] 0110 20:53:01.348871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:01.348945 (+ 74us) service_pool.cc:225] Handling call
0110 20:53:02.388266 (+1039321us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:01.349695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:01.349822 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:01.349826 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:53:01.349828 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:01.349904 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:01.349913 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:01.349916 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:01.349992 (+ 76us) tablet.cc:698] Row locks acquired
0110 20:53:01.349994 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:01.350036 (+ 42us) write_op.cc:270] Start()
0110 20:53:01.944191 (+594155us) write_op.cc:276] Timestamp: P: 1736542381349992 usec, L: 0
0110 20:53:01.944196 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:01.944395 (+ 199us) log.cc:844] Serialized 354 byte log entry
0110 20:53:02.386709 (+442314us) op_driver.cc:464] REPLICATION: finished
0110 20:53:02.387714 (+ 1005us) write_op.cc:301] APPLY: starting
0110 20:53:02.387732 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:02.387772 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:02.387833 (+ 61us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:02.387935 (+ 102us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:02.387968 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:02.387975 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:02.388051 (+ 76us) log.cc:844] Serialized 22 byte log entry
0110 20:53:02.388129 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:02.388147 (+ 18us) write_op.cc:454] Released schema lock
0110 20:53:02.388155 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":912,"num_ops":1,"prepare.queue_time_us":291,"prepare.run_cpu_time_us":1040,"prepare.run_wall_time_us":595110,"raft.queue_time_us":191,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":442474,"spinlock_wait_cycles":48384,"thread_start_us":1248,"threads_started":3}]]}
I20250110 20:53:02.555873 4133 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:02.556092 4133 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:02.556888 4133 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:02.557364 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:02.557655 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:02.558178 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:02.558809 4133 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:02.726913 4116 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:02.727124 4116 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:02.727954 4116 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:02.728351 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:02.728677 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:02.728933 4136 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=13, attempt_no=0}) took 1353 ms. Trace:
I20250110 20:53:02.729041 4136 rpcz_store.cc:276] 0110 20:53:01.375120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:01.375182 (+ 62us) service_pool.cc:225] Handling call
0110 20:53:02.728867 (+1353685us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:01.944836 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:01.944949 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:01.944955 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:53:01.944957 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:01.945035 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:01.945047 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:01.945052 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:01.945106 (+ 54us) tablet.cc:698] Row locks acquired
0110 20:53:01.945109 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:01.945158 (+ 49us) write_op.cc:270] Start()
0110 20:53:02.699328 (+754170us) write_op.cc:276] Timestamp: P: 1736542381945092 usec, L: 0
0110 20:53:02.699334 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:02.699561 (+ 227us) log.cc:844] Serialized 354 byte log entry
0110 20:53:02.728142 (+ 28581us) op_driver.cc:464] REPLICATION: finished
0110 20:53:02.728259 (+ 117us) write_op.cc:301] APPLY: starting
0110 20:53:02.728279 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:02.728323 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:02.728360 (+ 37us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:02.728437 (+ 77us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:02.728471 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:02.728525 (+ 54us) write_op.cc:312] APPLY: finished
0110 20:53:02.728634 (+ 109us) log.cc:844] Serialized 22 byte log entry
0110 20:53:02.728742 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:02.728763 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:02.728771 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"mutex_wait_us":27,"num_ops":1,"prepare.queue_time_us":569197,"prepare.run_cpu_time_us":1216,"prepare.run_wall_time_us":755300,"raft.queue_time_us":139,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":227,"replication_time_us":28770,"spinlock_wait_cycles":3456,"thread_start_us":379,"threads_started":1}]]}
I20250110 20:53:02.729470 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:02.729875 4116 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:02.909898 4133 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:02.910092 4133 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:02.910843 4133 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:02.911648 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:02.911140 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:02.912411 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:02.912691 4133 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:02.927663 4116 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:02.927873 4116 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:02.928848 4116 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:02.931250 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:02.931861 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:02.932647 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:02.932956 4116 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:03.466540 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:03.466892 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:03.467845 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:03.467526 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:03.471181 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:03.471758 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:03.472456 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:04.372421 4143 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:04.372668 4143 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:04.373436 4143 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:04.373968 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:04.374022 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:04.374630 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:04.374922 4143 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:04.375881 4148 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=14, attempt_no=0}) took 1985 ms. Trace:
I20250110 20:53:04.375994 4148 rpcz_store.cc:276] 0110 20:53:02.390227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:02.390298 (+ 71us) service_pool.cc:225] Handling call
0110 20:53:04.375825 (+1985527us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:02.700165 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:02.700282 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:02.700287 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:02.700289 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:02.700368 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:02.700378 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:02.700381 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:02.700426 (+ 45us) tablet.cc:698] Row locks acquired
0110 20:53:02.700428 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:02.700470 (+ 42us) write_op.cc:270] Start()
0110 20:53:03.594625 (+894155us) write_op.cc:276] Timestamp: P: 1736542382700429 usec, L: 0
0110 20:53:03.594630 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:03.594891 (+ 261us) log.cc:844] Serialized 354 byte log entry
0110 20:53:04.374652 (+779761us) op_driver.cc:464] REPLICATION: finished
0110 20:53:04.375306 (+ 654us) write_op.cc:301] APPLY: starting
0110 20:53:04.375322 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:04.375359 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:04.375378 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:04.375446 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:04.375472 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:04.375477 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:53:04.375586 (+ 109us) log.cc:844] Serialized 22 byte log entry
0110 20:53:04.375691 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:04.375711 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:04.375719 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":587,"num_ops":1,"prepare.queue_time_us":309417,"prepare.run_cpu_time_us":744,"prepare.run_wall_time_us":894801,"replication_time_us":779987,"thread_start_us":527,"threads_started":1}]]}
I20250110 20:53:04.419735 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:04.419948 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:04.420652 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:04.421123 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:04.421151 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 17 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:04.421779 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:05.268954 4143 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:05.269146 4143 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:05.269822 4143 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:05.270370 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:05.270402 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:05.271128 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:05.271697 4157 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=15, attempt_no=0}) took 2540 ms. Trace:
I20250110 20:53:05.271842 4143 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:05.271848 4157 rpcz_store.cc:276] 0110 20:53:02.731005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:02.731068 (+ 63us) service_pool.cc:225] Handling call
0110 20:53:05.271640 (+2540572us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:03.595018 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:03.595151 (+ 133us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:03.595160 (+ 9us) write_op.cc:435] Acquired schema lock
0110 20:53:03.595162 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:03.595262 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:03.595274 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:03.595279 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:03.595334 (+ 55us) tablet.cc:698] Row locks acquired
0110 20:53:03.595336 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:03.595386 (+ 50us) write_op.cc:270] Start()
0110 20:53:03.774546 (+179160us) write_op.cc:276] Timestamp: P: 1736542383595358 usec, L: 0
0110 20:53:03.774552 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:03.774807 (+ 255us) log.cc:844] Serialized 354 byte log entry
0110 20:53:05.270458 (+1495651us) op_driver.cc:464] REPLICATION: finished
0110 20:53:05.271012 (+ 554us) write_op.cc:301] APPLY: starting
0110 20:53:05.271032 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:05.271077 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:05.271133 (+ 56us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:05.271213 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:05.271247 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:05.271254 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:05.271370 (+ 116us) log.cc:844] Serialized 22 byte log entry
0110 20:53:05.271477 (+ 107us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:05.271508 (+ 31us) write_op.cc:454] Released schema lock
0110 20:53:05.271517 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":472,"num_ops":1,"prepare.queue_time_us":863380,"prepare.run_cpu_time_us":803,"prepare.run_wall_time_us":179865,"replication_time_us":1495875,"thread_start_us":389,"threads_started":1}]]}
I20250110 20:53:05.384603 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:05.443073 4143 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:05.443290 4143 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:05.443944 4143 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:05.444499 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:05.444521 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 18 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:05.445041 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:05.445322 4143 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:06.738932 4143 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:06.739104 4143 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:06.739923 4143 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:06.740327 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:06.740381 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:06.741058 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:06.741515 4143 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:06.742237 4166 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=16, attempt_no=0}) took 2364 ms. Trace:
I20250110 20:53:06.742333 4166 rpcz_store.cc:276] 0110 20:53:04.377765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:04.377832 (+ 67us) service_pool.cc:225] Handling call
0110 20:53:06.742194 (+2364362us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:04.378562 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:04.378678 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:04.378683 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:04.378684 (+ 1us) tablet.cc:659] Decoding operations
0110 20:53:04.378787 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:04.378802 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:04.378806 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:05.271538 (+892732us) lock_manager.cc:598] Waited 892606us for lock on <redacted> (key hash 2331656000287100367)
0110 20:53:05.271550 (+ 12us) tablet.cc:698] Row locks acquired
0110 20:53:05.271553 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:05.271659 (+ 106us) write_op.cc:270] Start()
0110 20:53:05.797796 (+526137us) write_op.cc:276] Timestamp: P: 1736542385271621 usec, L: 0
0110 20:53:05.797802 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:05.798003 (+ 201us) log.cc:844] Serialized 354 byte log entry
0110 20:53:06.741087 (+943084us) op_driver.cc:464] REPLICATION: finished
0110 20:53:06.741553 (+ 466us) write_op.cc:301] APPLY: starting
0110 20:53:06.741629 (+ 76us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:06.741674 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:06.741676 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:06.741774 (+ 98us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:06.741800 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:06.741805 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:53:06.741911 (+ 106us) log.cc:844] Serialized 43 byte log entry
0110 20:53:06.742000 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:06.742021 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:06.742029 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":373,"num_ops":1,"prepare.queue_time_us":310,"prepare.run_cpu_time_us":883,"prepare.run_wall_time_us":1419514,"replication_time_us":943229,"row_lock_wait_count":1,"row_lock_wait_us":892606,"spinlock_wait_cycles":101632,"thread_start_us":479,"threads_started":2}]]}
I20250110 20:53:06.796250 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:06.796466 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:06.797255 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:06.797678 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:06.797706 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 19 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:06.798265 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:06.798578 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:06.998193 4143 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:06.998389 4143 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:06.999150 4143 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:06.999693 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:06.999739 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:07.000360 4166 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=17, attempt_no=0}) took 1726 ms. Trace:
I20250110 20:53:07.000516 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:07.000799 4143 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:07.000494 4166 rpcz_store.cc:276] 0110 20:53:05.273637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:05.273720 (+ 83us) service_pool.cc:225] Handling call
0110 20:53:07.000287 (+1726567us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:05.798115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:05.798238 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:05.798245 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:05.798248 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:05.798349 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:05.798361 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:05.798366 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:05.798422 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:53:05.798425 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:05.798474 (+ 49us) write_op.cc:270] Start()
0110 20:53:06.672624 (+874150us) write_op.cc:276] Timestamp: P: 1736542385798435 usec, L: 0
0110 20:53:06.672629 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:06.672869 (+ 240us) log.cc:844] Serialized 354 byte log entry
0110 20:53:06.999143 (+326274us) op_driver.cc:464] REPLICATION: finished
0110 20:53:06.999259 (+ 116us) write_op.cc:301] APPLY: starting
0110 20:53:06.999277 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:06.999323 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:06.999352 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:06.999431 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:06.999464 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:06.999472 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:06.999585 (+ 113us) log.cc:844] Serialized 43 byte log entry
0110 20:53:07.000139 (+ 554us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:07.000159 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:07.000167 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":66,"num_ops":1,"prepare.queue_time_us":523939,"prepare.run_cpu_time_us":782,"prepare.run_wall_time_us":874842,"replication_time_us":326479}]]}
I20250110 20:53:07.343343 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:07.343526 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:07.344197 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:07.344765 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:07.345019 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 20 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:07.345532 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:07.389416 4143 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:07.389629 4143 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:07.390408 4143 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:07.390836 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 21 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:07.390832 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 21 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:07.391484 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:07.391917 4143 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:07.790372 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:08.429536 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:08.429780 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:08.430605 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:08.431136 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:08.431833 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:08.432528 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:08.432660 4179 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=19, attempt_no=0}) took 1430 ms. Trace:
I20250110 20:53:08.432843 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:08.432817 4179 rpcz_store.cc:276] 0110 20:53:07.002780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:07.002840 (+ 60us) service_pool.cc:225] Handling call
0110 20:53:08.432607 (+1429767us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:07.160455 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:07.160574 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:07.160581 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:07.160583 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:07.160677 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:07.160688 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:07.160694 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:07.160759 (+ 65us) tablet.cc:698] Row locks acquired
0110 20:53:07.160762 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:07.160810 (+ 48us) write_op.cc:270] Start()
0110 20:53:07.896956 (+736146us) write_op.cc:276] Timestamp: P: 1736542387160731 usec, L: 0
0110 20:53:07.896960 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:53:07.897183 (+ 223us) log.cc:844] Serialized 354 byte log entry
0110 20:53:08.431194 (+534011us) op_driver.cc:464] REPLICATION: finished
0110 20:53:08.431851 (+ 657us) write_op.cc:301] APPLY: starting
0110 20:53:08.431872 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:08.431917 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:08.431951 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:08.432072 (+ 121us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:08.432105 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:08.432112 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:08.432243 (+ 131us) log.cc:844] Serialized 43 byte log entry
0110 20:53:08.432320 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:08.432341 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:08.432349 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":568,"num_ops":1,"prepare.queue_time_us":157088,"prepare.run_cpu_time_us":744,"prepare.run_wall_time_us":736803,"replication_time_us":534158,"thread_start_us":434,"threads_started":1}]]}
I20250110 20:53:08.862095 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:08.862298 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:08.863157 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:08.863600 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:08.863662 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 22 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:08.864255 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:09.407930 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:09.408128 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:09.408979 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:09.409549 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 23 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:09.409816 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 23 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:09.410379 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:09.410974 4187 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=20, attempt_no=0}) took 2018 ms. Trace:
I20250110 20:53:09.411079 4187 rpcz_store.cc:276] 0110 20:53:07.392655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:07.392720 (+ 65us) service_pool.cc:225] Handling call
0110 20:53:09.410918 (+2018198us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:07.897305 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:07.897468 (+ 163us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:07.897476 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:53:07.897478 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:07.897576 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:07.897588 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:07.897593 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:07.897660 (+ 67us) tablet.cc:698] Row locks acquired
0110 20:53:07.897662 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:07.897713 (+ 51us) write_op.cc:270] Start()
0110 20:53:08.288880 (+391167us) write_op.cc:276] Timestamp: P: 1736542387897671 usec, L: 0
0110 20:53:08.288886 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:08.289119 (+ 233us) log.cc:844] Serialized 354 byte log entry
0110 20:53:09.409407 (+1120288us) op_driver.cc:464] REPLICATION: finished
0110 20:53:09.410256 (+ 849us) write_op.cc:301] APPLY: starting
0110 20:53:09.410274 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:09.410384 (+ 110us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:09.410415 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:09.410498 (+ 83us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:09.410530 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:09.410537 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:09.410650 (+ 113us) log.cc:844] Serialized 43 byte log entry
0110 20:53:09.410725 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:09.410768 (+ 43us) write_op.cc:454] Released schema lock
0110 20:53:09.410776 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":765,"num_ops":1,"prepare.queue_time_us":504108,"prepare.run_cpu_time_us":826,"prepare.run_wall_time_us":391888,"replication_time_us":1120486,"thread_start_us":676,"threads_started":1}]]}
I20250110 20:53:09.425310 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:09.891847 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:10.292415 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:10.292629 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:10.293594 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:10.294169 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:10.294195 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:10.294975 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:10.541805 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:10.541991 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:10.542804 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:10.543366 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:10.543493 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 25 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:10.544085 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:10.544346 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:10.909201 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:10.911612 4200 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=23, attempt_no=0}) took 1483 ms. Trace:
I20250110 20:53:10.911770 4200 rpcz_store.cc:276] 0110 20:53:09.428455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:09.428516 (+ 61us) service_pool.cc:225] Handling call
0110 20:53:10.911553 (+1483037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:09.623339 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:09.623471 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:09.623478 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:09.623481 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:09.623583 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:09.623595 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:09.623600 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:09.623670 (+ 70us) tablet.cc:698] Row locks acquired
0110 20:53:09.623672 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:09.623720 (+ 48us) write_op.cc:270] Start()
0110 20:53:10.255878 (+632158us) write_op.cc:276] Timestamp: P: 1736542389623649 usec, L: 0
0110 20:53:10.255884 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:10.256137 (+ 253us) log.cc:844] Serialized 354 byte log entry
0110 20:53:10.910504 (+654367us) op_driver.cc:464] REPLICATION: finished
0110 20:53:10.910889 (+ 385us) write_op.cc:301] APPLY: starting
0110 20:53:10.910906 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:10.910941 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:10.910974 (+ 33us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:10.911078 (+ 104us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:10.911111 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:10.911117 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:53:10.911200 (+ 83us) log.cc:844] Serialized 22 byte log entry
0110 20:53:10.911320 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:10.911456 (+ 136us) write_op.cc:454] Released schema lock
0110 20:53:10.911464 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":332,"num_ops":1,"prepare.queue_time_us":194303,"prepare.run_cpu_time_us":822,"prepare.run_wall_time_us":632881,"replication_time_us":654585,"thread_start_us":266,"threads_started":1}]]}
I20250110 20:53:11.248349 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:11.248623 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:11.249383 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:11.249863 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:11.249940 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 27 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:11.251286 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:11.251611 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:11.451989 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:11.452225 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:11.452983 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:11.453441 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:11.453755 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 26 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:11.454248 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:12.299950 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:12.300143 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:12.300832 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:12.301419 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:12.301422 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 28 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:12.302127 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:12.302815 4208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=25, attempt_no=0}) took 1389 ms. Trace:
I20250110 20:53:12.302930 4208 rpcz_store.cc:276] 0110 20:53:10.913607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:10.913666 (+ 59us) service_pool.cc:225] Handling call
0110 20:53:12.302732 (+1389066us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:11.039899 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:11.040045 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:11.040053 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:53:11.040055 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:11.040144 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:11.040154 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:11.040157 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:11.040203 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:53:11.040205 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:11.040252 (+ 47us) write_op.cc:270] Start()
0110 20:53:11.402432 (+362180us) write_op.cc:276] Timestamp: P: 1736542391040207 usec, L: 0
0110 20:53:11.402438 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:11.402702 (+ 264us) log.cc:844] Serialized 354 byte log entry
0110 20:53:12.301528 (+898826us) op_driver.cc:464] REPLICATION: finished
0110 20:53:12.302165 (+ 637us) write_op.cc:301] APPLY: starting
0110 20:53:12.302183 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:12.302225 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:12.302259 (+ 34us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:12.302344 (+ 85us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:12.302380 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:12.302386 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:53:12.302476 (+ 90us) log.cc:844] Serialized 22 byte log entry
0110 20:53:12.302564 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:12.302599 (+ 35us) write_op.cc:454] Released schema lock
0110 20:53:12.302608 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":573,"num_ops":1,"prepare.queue_time_us":125806,"prepare.run_cpu_time_us":840,"prepare.run_wall_time_us":362922,"raft.queue_time_us":39,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":209,"replication_time_us":899036,"spinlock_wait_cycles":64384,"thread_start_us":1233,"threads_started":1}]]}
I20250110 20:53:12.345459 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:12.402918 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:12.403110 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:12.403885 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:12.404423 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:12.404394 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 29 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:12.405026 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:12.405076 4208 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=26, attempt_no=0}) took 1153 ms. Trace:
I20250110 20:53:12.405157 4208 rpcz_store.cc:276] 0110 20:53:11.251936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:11.252004 (+ 68us) service_pool.cc:225] Handling call
0110 20:53:12.405030 (+1153026us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:11.402856 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:11.402984 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:11.402990 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:53:11.402993 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:11.403095 (+ 102us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:11.403106 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:11.403110 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:11.403157 (+ 47us) tablet.cc:698] Row locks acquired
0110 20:53:11.403159 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:11.403198 (+ 39us) write_op.cc:270] Start()
0110 20:53:11.522412 (+119214us) write_op.cc:276] Timestamp: P: 1736542391403164 usec, L: 0
0110 20:53:11.522418 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:11.522704 (+ 286us) log.cc:844] Serialized 354 byte log entry
0110 20:53:12.404120 (+881416us) op_driver.cc:464] REPLICATION: finished
0110 20:53:12.404316 (+ 196us) write_op.cc:301] APPLY: starting
0110 20:53:12.404362 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:12.404414 (+ 52us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:12.404444 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:12.404524 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:12.404556 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:12.404563 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:12.404666 (+ 103us) log.cc:844] Serialized 43 byte log entry
0110 20:53:12.404738 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:12.404898 (+ 160us) write_op.cc:454] Released schema lock
0110 20:53:12.404907 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":123,"num_ops":1,"prepare.queue_time_us":150347,"prepare.run_cpu_time_us":848,"prepare.run_wall_time_us":119948,"replication_time_us":881663}]]}
I20250110 20:53:12.405524 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:13.253973 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:13.668627 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:13.668833 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:13.669538 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:13.669965 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:13.670003 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 30 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:13.670643 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:13.670964 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:13.671314 4219 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=27, attempt_no=0}) took 1366 ms. Trace:
I20250110 20:53:13.671418 4219 rpcz_store.cc:276] 0110 20:53:12.304761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:12.304825 (+ 64us) service_pool.cc:225] Handling call
0110 20:53:13.671267 (+1366442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:12.305525 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:12.305632 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:12.305637 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:12.305639 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:12.305721 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:12.305731 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:12.305734 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:12.305798 (+ 64us) tablet.cc:698] Row locks acquired
0110 20:53:12.305799 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:53:12.305832 (+ 33us) write_op.cc:270] Start()
0110 20:53:12.738976 (+433144us) write_op.cc:276] Timestamp: P: 1736542392305811 usec, L: 0
0110 20:53:12.738982 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:12.739209 (+ 227us) log.cc:844] Serialized 354 byte log entry
0110 20:53:13.670120 (+930911us) op_driver.cc:464] REPLICATION: finished
0110 20:53:13.670632 (+ 512us) write_op.cc:301] APPLY: starting
0110 20:53:13.670652 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:13.670695 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:13.670781 (+ 86us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:13.670881 (+ 100us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:13.670915 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:13.670922 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:13.671060 (+ 138us) log.cc:844] Serialized 43 byte log entry
0110 20:53:13.671131 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:13.671152 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:13.671159 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":448,"num_ops":1,"prepare.queue_time_us":285,"prepare.run_cpu_time_us":715,"prepare.run_wall_time_us":433777,"replication_time_us":931103,"thread_start_us":660,"threads_started":2}]]}
I20250110 20:53:14.241863 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:14.242066 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:14.242951 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:14.243433 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:14.243543 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:14.244163 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:14.244441 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:14.245644 4225 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=28, attempt_no=0}) took 1838 ms. Trace:
I20250110 20:53:14.245779 4225 rpcz_store.cc:276] 0110 20:53:12.407006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:12.407075 (+ 69us) service_pool.cc:225] Handling call
0110 20:53:14.245589 (+1838514us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:12.739335 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:12.739451 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:12.739456 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:12.739458 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:12.739548 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:12.739560 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:12.739565 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:12.739621 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:53:12.739623 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:12.739670 (+ 47us) write_op.cc:270] Start()
0110 20:53:13.007824 (+268154us) write_op.cc:276] Timestamp: P: 1736542392739605 usec, L: 0
0110 20:53:13.007829 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:13.008083 (+ 254us) log.cc:844] Serialized 354 byte log entry
0110 20:53:14.244428 (+1236345us) op_driver.cc:464] REPLICATION: finished
0110 20:53:14.245001 (+ 573us) write_op.cc:301] APPLY: starting
0110 20:53:14.245019 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:14.245062 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:14.245093 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:14.245177 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:14.245210 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:14.245217 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:14.245346 (+ 129us) log.cc:844] Serialized 43 byte log entry
0110 20:53:14.245424 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:14.245444 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:14.245451 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":497,"num_ops":1,"prepare.queue_time_us":331780,"prepare.run_cpu_time_us":779,"prepare.run_wall_time_us":268842,"replication_time_us":1236566,"thread_start_us":419,"threads_started":1}]]}
I20250110 20:53:14.622879 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:14.623085 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:14.623904 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:14.624444 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:14.624467 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 31 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:14.625051 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:14.660928 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:14.661149 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:14.662065 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:14.662621 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:14.662617 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:14.663328 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:14.663723 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:14.801949 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:14.935082 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:14.935272 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:14.936031 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:14.936560 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:14.937404 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:14.937870 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:14.938254 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:15.017159 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:15.017378 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:15.018167 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:15.018898 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:15.019306 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:15.019732 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:15.019997 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:15.027499 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:15.027704 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:15.028342 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:15.028755 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:15.028796 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 32 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:15.029631 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:15.029893 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:15.636353 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:15.636544 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:15.637362 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:15.637866 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:15.637972 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:15.638501 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:15.638870 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:15.640093 4235 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=30, attempt_no=0}) took 1392 ms. Trace:
I20250110 20:53:15.640213 4235 rpcz_store.cc:276] 0110 20:53:14.247769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:14.247831 (+ 62us) service_pool.cc:225] Handling call
0110 20:53:15.640037 (+1392206us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:14.248548 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:14.248642 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:14.248646 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:53:14.248648 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:14.248726 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:14.248737 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:14.248741 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:14.248792 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:53:14.248794 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:14.248829 (+ 35us) write_op.cc:270] Start()
0110 20:53:15.055963 (+807134us) write_op.cc:276] Timestamp: P: 1736542394248764 usec, L: 0
0110 20:53:15.055968 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:15.056160 (+ 192us) log.cc:844] Serialized 354 byte log entry
0110 20:53:15.638714 (+582554us) op_driver.cc:464] REPLICATION: finished
0110 20:53:15.639394 (+ 680us) write_op.cc:301] APPLY: starting
0110 20:53:15.639413 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:15.639446 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:15.639473 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:15.639557 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:15.639581 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:15.639586 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:53:15.639673 (+ 87us) log.cc:844] Serialized 43 byte log entry
0110 20:53:15.639874 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:15.639896 (+ 22us) write_op.cc:454] Released schema lock
0110 20:53:15.639904 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":579,"mutex_wait_us":34,"num_ops":1,"prepare.queue_time_us":298,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":807710,"raft.queue_time_us":82,"raft.run_cpu_time_us":549,"raft.run_wall_time_us":592,"replication_time_us":582709,"spinlock_wait_cycles":135808,"thread_start_us":724,"threads_started":2}]]}
I20250110 20:53:15.668483 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:15.668702 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:15.669481 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:15.669824 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:15.670162 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 33 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:15.670851 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:15.671358 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:16.381639 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:16.381846 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:16.382609 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:16.383149 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:16.383222 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:16.383809 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:16.384094 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:16.384657 4244 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=31, attempt_no=0}) took 1720 ms. Trace:
I20250110 20:53:16.384776 4244 rpcz_store.cc:276] 0110 20:53:14.664182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:14.664252 (+ 70us) service_pool.cc:225] Handling call
0110 20:53:16.384608 (+1720356us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:15.056300 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:15.056467 (+ 167us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:15.056475 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:53:15.056477 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:15.056585 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:15.056597 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:15.056603 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:15.056656 (+ 53us) tablet.cc:698] Row locks acquired
0110 20:53:15.056658 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:15.056708 (+ 50us) write_op.cc:270] Start()
0110 20:53:15.828849 (+772141us) write_op.cc:276] Timestamp: P: 1736542395056665 usec, L: 0
0110 20:53:15.828854 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:15.829091 (+ 237us) log.cc:844] Serialized 354 byte log entry
0110 20:53:16.383350 (+554259us) op_driver.cc:464] REPLICATION: finished
0110 20:53:16.383988 (+ 638us) write_op.cc:301] APPLY: starting
0110 20:53:16.384007 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:16.384053 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:16.384093 (+ 40us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:16.384180 (+ 87us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:16.384226 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:16.384234 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:16.384331 (+ 97us) log.cc:844] Serialized 22 byte log entry
0110 20:53:16.384442 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:16.384463 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:16.384471 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":570,"num_ops":1,"prepare.queue_time_us":391540,"prepare.run_cpu_time_us":820,"prepare.run_wall_time_us":772880,"raft.queue_time_us":300,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":212,"replication_time_us":554443,"spinlock_wait_cycles":10496,"thread_start_us":494,"threads_started":1}]]}
I20250110 20:53:16.771638 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:16.771837 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:16.772634 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:16.773161 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:16.773200 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:16.773867 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:17.283658 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:17.286845 4251 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=32, attempt_no=0}) took 1644 ms. Trace:
I20250110 20:53:17.287073 4251 rpcz_store.cc:276] 0110 20:53:15.642005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:15.642077 (+ 72us) service_pool.cc:225] Handling call
0110 20:53:17.286788 (+1644711us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:15.829212 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:15.829335 (+ 123us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:15.829342 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:15.829345 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:15.829442 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:15.829454 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:15.829459 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:15.829512 (+ 53us) tablet.cc:698] Row locks acquired
0110 20:53:15.829515 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:15.829567 (+ 52us) write_op.cc:270] Start()
0110 20:53:15.953768 (+124201us) write_op.cc:276] Timestamp: P: 1736542395829497 usec, L: 0
0110 20:53:15.953773 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:15.953997 (+ 224us) log.cc:844] Serialized 354 byte log entry
0110 20:53:17.285110 (+1331113us) op_driver.cc:464] REPLICATION: finished
0110 20:53:17.285482 (+ 372us) write_op.cc:301] APPLY: starting
0110 20:53:17.285501 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:17.285544 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:17.286216 (+ 672us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:17.286353 (+ 137us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:17.286379 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:17.286385 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:53:17.286513 (+ 128us) log.cc:844] Serialized 43 byte log entry
0110 20:53:17.286598 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:17.286635 (+ 37us) write_op.cc:454] Released schema lock
0110 20:53:17.286641 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":284,"num_ops":1,"prepare.queue_time_us":186737,"prepare.run_cpu_time_us":786,"prepare.run_wall_time_us":124858,"replication_time_us":1331306,"thread_start_us":204,"threads_started":1}]]}
I20250110 20:53:17.366847 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:17.367044 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:17.367858 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:17.368115 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 35 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:17.368641 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 35 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:17.369275 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:17.649583 4174 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:17.649785 4174 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:17.650538 4174 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:17.651217 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:17.651228 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:17.652379 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:17.652669 4174 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:17.654110 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:18.333895 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:18.334075 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:18.334918 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:18.335656 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:18.335701 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:18.336265 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:18.336697 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:18.467896 4261 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:18.468214 4261 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:18.468966 4261 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:18.469342 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:18.469522 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 37 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:18.470240 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:18.470590 4261 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:18.554384 4261 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:18.554596 4261 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:18.555356 4261 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:18.555897 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:18.555990 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:18.556651 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:18.557093 4263 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=35, attempt_no=0}) took 1268 ms. Trace:
I20250110 20:53:18.557209 4263 rpcz_store.cc:276] 0110 20:53:17.289044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:17.289096 (+ 52us) service_pool.cc:225] Handling call
0110 20:53:18.557049 (+1267953us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:17.548862 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:17.548963 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:17.548968 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:17.548970 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:17.549048 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:17.549060 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:17.549064 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:17.549125 (+ 61us) tablet.cc:698] Row locks acquired
0110 20:53:17.549128 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:17.549176 (+ 48us) write_op.cc:270] Start()
0110 20:53:18.528345 (+979169us) write_op.cc:276] Timestamp: P: 1736542397549138 usec, L: 0
0110 20:53:18.528351 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:18.528536 (+ 185us) log.cc:844] Serialized 354 byte log entry
0110 20:53:18.555694 (+ 27158us) op_driver.cc:464] REPLICATION: finished
0110 20:53:18.556224 (+ 530us) write_op.cc:301] APPLY: starting
0110 20:53:18.556242 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:18.556292 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:18.556457 (+ 165us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:18.556582 (+ 125us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:18.556615 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:18.556622 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:18.556791 (+ 169us) log.cc:844] Serialized 43 byte log entry
0110 20:53:18.556859 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:18.556879 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:18.556886 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":440,"num_ops":1,"prepare.queue_time_us":259378,"prepare.run_cpu_time_us":664,"prepare.run_wall_time_us":979732,"replication_time_us":27192,"thread_start_us":309,"threads_started":1}]]}
I20250110 20:53:18.557618 4261 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:18.590683 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:18.590907 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:18.591720 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:18.592247 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:18.592288 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 38 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:18.592882 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:18.593269 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:19.114113 4261 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:19.114315 4261 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:19.115083 4261 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:19.115556 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:19.115530 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:19.116353 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:19.116840 4261 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:19.116873 4266 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=36, attempt_no=0}) took 1463 ms. Trace:
I20250110 20:53:19.117081 4266 rpcz_store.cc:276] 0110 20:53:17.653180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:17.653227 (+ 47us) service_pool.cc:225] Handling call
0110 20:53:19.116820 (+1463593us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:18.528630 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:18.528778 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:18.528785 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:18.528788 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:18.528882 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:18.528892 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:18.528895 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:18.528939 (+ 44us) tablet.cc:698] Row locks acquired
0110 20:53:18.528941 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:18.528989 (+ 48us) write_op.cc:270] Start()
0110 20:53:18.763137 (+234148us) write_op.cc:276] Timestamp: P: 1736542398528957 usec, L: 0
0110 20:53:18.763142 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:18.763362 (+ 220us) log.cc:844] Serialized 354 byte log entry
0110 20:53:19.115519 (+352157us) op_driver.cc:464] REPLICATION: finished
0110 20:53:19.116275 (+ 756us) write_op.cc:301] APPLY: starting
0110 20:53:19.116294 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:19.116341 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:19.116372 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:19.116447 (+ 75us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:19.116479 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:19.116485 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:53:19.116594 (+ 109us) log.cc:844] Serialized 43 byte log entry
0110 20:53:19.116663 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:19.116684 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:19.116690 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":661,"num_ops":1,"prepare.queue_time_us":875004,"prepare.run_cpu_time_us":735,"prepare.run_wall_time_us":234804,"replication_time_us":351895,"thread_start_us":538,"threads_started":1}]]}
I20250110 20:53:19.542790 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:19.543001 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:19.543814 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:19.544343 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:19.544620 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 39 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:19.545178 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:19.598565 4261 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:19.598798 4261 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:19.599483 4261 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:19.600167 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:19.600314 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:19.600728 4266 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=37, attempt_no=0}) took 1041 ms. Trace:
I20250110 20:53:19.600947 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:19.600862 4266 rpcz_store.cc:276] 0110 20:53:18.559082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:18.559149 (+ 67us) service_pool.cc:225] Handling call
0110 20:53:19.600659 (+1041510us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:18.763485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:18.763634 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:18.763642 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:53:18.763645 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:18.763740 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:18.763752 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:18.763756 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:18.763812 (+ 56us) tablet.cc:698] Row locks acquired
0110 20:53:18.763814 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:18.763863 (+ 49us) write_op.cc:270] Start()
0110 20:53:19.436016 (+672153us) write_op.cc:276] Timestamp: P: 1736542398763808 usec, L: 0
0110 20:53:19.436022 (+ 6us) op_driver.cc:348] REPLICATION: starting
0110 20:53:19.436248 (+ 226us) log.cc:844] Serialized 354 byte log entry
0110 20:53:19.599973 (+163725us) op_driver.cc:464] REPLICATION: finished
0110 20:53:19.600093 (+ 120us) write_op.cc:301] APPLY: starting
0110 20:53:19.600136 (+ 43us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:19.600173 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:19.600203 (+ 30us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:19.600300 (+ 97us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:19.600332 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:19.600339 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:19.600448 (+ 109us) log.cc:844] Serialized 43 byte log entry
0110 20:53:19.600509 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:19.600530 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:19.600537 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":62,"num_ops":1,"prepare.queue_time_us":203900,"prepare.run_cpu_time_us":774,"prepare.run_wall_time_us":672841,"replication_time_us":163915}]]}
I20250110 20:53:19.601953 4261 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:19.696481 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:19.804862 4261 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:19.805063 4261 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:19.805842 4261 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:19.806308 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:19.806313 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:19.806974 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:19.807282 4261 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:20.194159 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:20.194416 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:20.195356 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:20.195900 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:20.195581 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 41 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:20.196942 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:20.197232 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:20.655329 4142 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:20.655537 4142 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:20.656301 4142 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:20.656833 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:20.656937 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:20.657636 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:20.658105 4142 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:20.658349 4271 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=39, attempt_no=0}) took 1055 ms. Trace:
I20250110 20:53:20.658607 4271 rpcz_store.cc:276] 0110 20:53:19.602798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:19.602846 (+ 48us) service_pool.cc:225] Handling call
0110 20:53:20.658285 (+1055439us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:19.662449 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:19.662579 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:19.662586 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:19.662587 (+ 1us) tablet.cc:659] Decoding operations
0110 20:53:19.662667 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:19.662677 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:19.662681 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:19.662727 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:53:19.662729 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:19.662818 (+ 89us) write_op.cc:270] Start()
0110 20:53:20.306994 (+644176us) write_op.cc:276] Timestamp: P: 1736542399662782 usec, L: 0
0110 20:53:20.307001 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:53:20.307244 (+ 243us) log.cc:844] Serialized 354 byte log entry
0110 20:53:20.656454 (+349210us) op_driver.cc:464] REPLICATION: finished
0110 20:53:20.657666 (+ 1212us) write_op.cc:301] APPLY: starting
0110 20:53:20.657686 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:20.657731 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:20.657786 (+ 55us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:20.657866 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:20.657899 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:20.657905 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:53:20.658013 (+ 108us) log.cc:844] Serialized 43 byte log entry
0110 20:53:20.658096 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:20.658117 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:20.658125 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1133,"num_ops":1,"prepare.queue_time_us":59165,"prepare.run_cpu_time_us":875,"prepare.run_wall_time_us":644927,"raft.queue_time_us":621,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":291,"replication_time_us":349410,"spinlock_wait_cycles":4992,"thread_start_us":1393,"threads_started":1}]]}
I20250110 20:53:21.217078 4270 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:21.217278 4270 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:21.218044 4270 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:21.218696 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:21.218724 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 42 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:21.219523 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:21.512331 4270 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:21.514397 4277 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=40, attempt_no=0}) took 1706 ms. Trace:
I20250110 20:53:21.514585 4277 rpcz_store.cc:276] 0110 20:53:19.808158 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:19.808233 (+ 75us) service_pool.cc:225] Handling call
0110 20:53:21.514336 (+1706103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:20.307416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:20.307565 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:20.307572 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:20.307575 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:20.307710 (+ 135us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:20.307721 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:20.307725 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:20.307771 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:53:20.307772 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:53:20.307810 (+ 38us) write_op.cc:270] Start()
0110 20:53:21.213975 (+906165us) write_op.cc:276] Timestamp: P: 1736542400307749 usec, L: 0
0110 20:53:21.213980 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:21.214216 (+ 236us) log.cc:844] Serialized 354 byte log entry
0110 20:53:21.513279 (+299063us) op_driver.cc:464] REPLICATION: finished
0110 20:53:21.513707 (+ 428us) write_op.cc:301] APPLY: starting
0110 20:53:21.513722 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:21.513769 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:21.513801 (+ 32us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:21.513909 (+ 108us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:21.513942 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:21.513950 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:21.514056 (+ 106us) log.cc:844] Serialized 43 byte log entry
0110 20:53:21.514136 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:21.514157 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:21.514165 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":362,"num_ops":1,"prepare.queue_time_us":498672,"prepare.run_cpu_time_us":1298,"prepare.run_wall_time_us":907370,"raft.queue_time_us":168,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":3,"replication_time_us":299266,"thread_start_us":698,"threads_started":2}]]}
I20250110 20:53:21.765368 4276 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:21.765558 4276 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:21.766443 4276 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:21.766999 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:21.767102 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:21.767771 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:21.768103 4276 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:22.017899 4282 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:22.018384 4282 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:22.019188 4282 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:22.019616 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:22.019975 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 43 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:22.020483 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:22.021121 4282 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:22.395990 4276 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:22.396174 4276 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:22.396855 4276 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:22.397361 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:22.397375 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:22.397976 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:22.398528 4276 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:22.398691 4284 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=41, attempt_no=0}) took 1738 ms. Trace:
I20250110 20:53:22.398877 4284 rpcz_store.cc:276] 0110 20:53:20.660485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:20.660570 (+ 85us) service_pool.cc:225] Handling call
0110 20:53:22.398623 (+1738053us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:21.214827 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:21.214940 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:21.214945 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:21.214947 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:21.215024 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:21.215034 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:21.215037 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:21.215083 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:53:21.215085 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:21.215127 (+ 42us) write_op.cc:270] Start()
0110 20:53:22.157247 (+942120us) write_op.cc:276] Timestamp: P: 1736542401215083 usec, L: 0
0110 20:53:22.157251 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:53:22.157453 (+ 202us) log.cc:844] Serialized 354 byte log entry
0110 20:53:22.397153 (+239700us) op_driver.cc:464] REPLICATION: finished
0110 20:53:22.397645 (+ 492us) write_op.cc:301] APPLY: starting
0110 20:53:22.397663 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:22.397889 (+ 226us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:22.397920 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:22.398043 (+ 123us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:22.398076 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:22.398083 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:22.398252 (+ 169us) log.cc:844] Serialized 43 byte log entry
0110 20:53:22.398325 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:22.398458 (+ 133us) write_op.cc:454] Released schema lock
0110 20:53:22.398465 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":419,"num_ops":1,"prepare.queue_time_us":553711,"prepare.run_cpu_time_us":661,"prepare.run_wall_time_us":942715,"raft.queue_time_us":79,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"replication_time_us":239864,"thread_start_us":315,"threads_started":1}]]}
I20250110 20:53:22.454501 4282 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:22.454691 4282 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:22.455468 4282 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:22.456084 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:22.456101 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 44 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:22.456773 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:22.457372 4282 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:23.560425 4290 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:23.560631 4290 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:23.561501 4290 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:23.561902 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:23.561935 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:23.562585 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:23.563467 4293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=42, attempt_no=0}) took 2046 ms. Trace:
I20250110 20:53:23.563638 4293 rpcz_store.cc:276] 0110 20:53:21.516496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:21.516560 (+ 64us) service_pool.cc:225] Handling call
0110 20:53:23.563411 (+2046851us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:22.157568 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:22.157706 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:22.157713 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:22.157716 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:22.157814 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:22.157833 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:22.157837 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:22.398361 (+240524us) lock_manager.cc:598] Waited 240449us for lock on <redacted> (key hash 16057051314367886621)
0110 20:53:22.398371 (+ 10us) tablet.cc:698] Row locks acquired
0110 20:53:22.398374 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:22.398423 (+ 49us) write_op.cc:270] Start()
0110 20:53:23.263553 (+865130us) write_op.cc:276] Timestamp: P: 1736542402398365 usec, L: 0
0110 20:53:23.263558 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:23.263755 (+ 197us) log.cc:844] Serialized 354 byte log entry
0110 20:53:23.561835 (+298080us) op_driver.cc:464] REPLICATION: finished
0110 20:53:23.562782 (+ 947us) write_op.cc:301] APPLY: starting
0110 20:53:23.562826 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:23.562870 (+ 44us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:23.562873 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:23.562992 (+ 119us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:23.563024 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:23.563031 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:23.563167 (+ 136us) log.cc:844] Serialized 43 byte log entry
0110 20:53:23.563226 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:23.563242 (+ 16us) write_op.cc:454] Released schema lock
0110 20:53:23.563248 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":823,"num_ops":1,"prepare.queue_time_us":640515,"prepare.run_cpu_time_us":763,"prepare.run_wall_time_us":1106252,"replication_time_us":298241,"row_lock_wait_count":1,"row_lock_wait_us":240449,"thread_start_us":733,"threads_started":1}]]}
I20250110 20:53:23.698832 4290 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:23.700428 4293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=43, attempt_no=0}) took 1299 ms. Trace:
I20250110 20:53:23.700546 4293 rpcz_store.cc:276] 0110 20:53:22.400888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:22.400963 (+ 75us) service_pool.cc:225] Handling call
0110 20:53:23.700371 (+1299408us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:23.263861 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:23.263986 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:23.263993 (+ 7us) write_op.cc:435] Acquired schema lock
0110 20:53:23.263996 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:23.264122 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:23.264134 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:23.264139 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:23.264198 (+ 59us) tablet.cc:698] Row locks acquired
0110 20:53:23.264201 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:23.264252 (+ 51us) write_op.cc:270] Start()
0110 20:53:23.494402 (+230150us) write_op.cc:276] Timestamp: P: 1736542403264195 usec, L: 0
0110 20:53:23.494407 (+ 5us) op_driver.cc:348] REPLICATION: starting
0110 20:53:23.494660 (+ 253us) log.cc:844] Serialized 354 byte log entry
0110 20:53:23.699729 (+205069us) op_driver.cc:464] REPLICATION: finished
0110 20:53:23.699837 (+ 108us) write_op.cc:301] APPLY: starting
0110 20:53:23.699855 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:23.699893 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:23.699918 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:23.699993 (+ 75us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:23.700026 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:23.700033 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:23.700162 (+ 129us) log.cc:844] Serialized 43 byte log entry
0110 20:53:23.700235 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:23.700251 (+ 16us) write_op.cc:454] Released schema lock
0110 20:53:23.700258 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":40,"num_ops":1,"prepare.queue_time_us":862368,"prepare.run_cpu_time_us":815,"prepare.run_wall_time_us":230876,"replication_time_us":205290}]]}
I20250110 20:53:23.839381 4289 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:23.839618 4289 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:23.840461 4289 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:23.840734 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:23.840950 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 45 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:23.841686 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:24.679271 4289 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:24.981841 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:24.982101 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:24.983016 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:24.983593 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:24.983644 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:24.984246 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:24.984477 4304 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=44, attempt_no=0}) took 1419 ms. Trace:
I20250110 20:53:24.984596 4304 rpcz_store.cc:276] 0110 20:53:23.565340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:23.565404 (+ 64us) service_pool.cc:225] Handling call
0110 20:53:24.984424 (+1419020us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:23.565904 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:23.565998 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:23.566002 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:53:23.566004 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:23.566074 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:23.566083 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:23.566086 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:23.566135 (+ 49us) tablet.cc:698] Row locks acquired
0110 20:53:23.566137 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:23.566169 (+ 32us) write_op.cc:270] Start()
0110 20:53:24.373306 (+807137us) write_op.cc:276] Timestamp: P: 1736542403566155 usec, L: 0
0110 20:53:24.373310 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:53:24.373516 (+ 206us) log.cc:844] Serialized 354 byte log entry
0110 20:53:24.983029 (+609513us) op_driver.cc:464] REPLICATION: finished
0110 20:53:24.983463 (+ 434us) write_op.cc:301] APPLY: starting
0110 20:53:24.983482 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:24.983525 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:24.983547 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:24.983764 (+ 217us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:24.983796 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:24.983803 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:24.983971 (+ 168us) log.cc:844] Serialized 43 byte log entry
0110 20:53:24.984237 (+ 266us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:24.984268 (+ 31us) write_op.cc:454] Released schema lock
0110 20:53:24.984276 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":358,"num_ops":1,"prepare.queue_time_us":57,"prepare.run_cpu_time_us":1970,"prepare.run_wall_time_us":809035,"raft.queue_time_us":141,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":609675,"thread_start_us":2750,"threads_started":2}]]}
I20250110 20:53:25.124907 4289 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:25.125141 4289 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.126003 4289 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.126433 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.126485 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 47 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.127095 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.229866 4299 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:25.231551 4304 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:48626 (ReqId={client: 93ac347173ab45e683f8e7b5c7e028ef, seq_no=45, attempt_no=0}) took 1529 ms. Trace:
I20250110 20:53:25.231664 4304 rpcz_store.cc:276] 0110 20:53:23.702451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:23.702515 (+ 64us) service_pool.cc:225] Handling call
0110 20:53:25.231488 (+1528973us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:24.374971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 26db1d18dd4541db812c55f643f305b3
0110 20:53:24.375074 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:24.375079 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:24.375081 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:24.375158 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:24.375167 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:24.375171 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:53:24.375217 (+ 46us) tablet.cc:698] Row locks acquired
0110 20:53:24.375218 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:53:24.375260 (+ 42us) write_op.cc:270] Start()
0110 20:53:24.878450 (+503190us) write_op.cc:276] Timestamp: P: 1736542404375211 usec, L: 0
0110 20:53:24.878457 (+ 7us) op_driver.cc:348] REPLICATION: starting
0110 20:53:24.878690 (+ 233us) log.cc:844] Serialized 354 byte log entry
0110 20:53:25.230836 (+352146us) op_driver.cc:464] REPLICATION: finished
0110 20:53:25.230928 (+ 92us) write_op.cc:301] APPLY: starting
0110 20:53:25.230946 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:25.230992 (+ 46us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:25.231046 (+ 54us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:25.231129 (+ 83us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:25.231159 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:25.231164 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:53:25.231266 (+ 102us) log.cc:844] Serialized 43 byte log entry
0110 20:53:25.231340 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:25.231361 (+ 21us) write_op.cc:454] Released schema lock
0110 20:53:25.231367 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":37,"num_ops":1,"prepare.queue_time_us":671973,"prepare.run_cpu_time_us":747,"prepare.run_wall_time_us":503824,"replication_time_us":352351}]]}
I20250110 20:53:25.289939 4289 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:25.335470 3693 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:53:25.345013 4289 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:25.345222 4289 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.345791 4289 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.346163 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.347132 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.347843 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.348297 4289 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:25.348371 3954 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:53:25.366222 3824 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:53:25.373948 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:25.374169 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.375195 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.376091 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.376441 3843 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 2.
I20250110 20:53:25.376654 4289 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:25.376817 4289 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.377530 4289 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.378166 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b; no voters:
I20250110 20:53:25.378371 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.378504 4299 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:53:25.378610 3712 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 2.
I20250110 20:53:25.378808 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:25.378944 4299 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:25.379590 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.381357 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.382208 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b; no voters: ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.382526 4289 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:53:25.382622 4289 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Starting leader election (detected failure of leader ca06c6ed33684aafbaebc0ddd43546f2)
I20250110 20:53:25.382704 4289 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:25.384132 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.384549 3713 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Rejecting Update request from peer ca06c6ed33684aafbaebc0ddd43546f2 for earlier term 2. Current term is 3. Ops: []
I20250110 20:53:25.385026 4289 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.385639 4289 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.385586 4288 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 }, Status: INVALID_TERM, Last received: 2.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.385905 4288 raft_consensus.cc:3049] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 LEADER]: Stepping down as leader of term 2
I20250110 20:53:25.386029 4288 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca06c6ed33684aafbaebc0ddd43546f2, State: Running, Role: LEADER
I20250110 20:53:25.386086 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.386158 3843 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 FOLLOWER]: Rejecting Update request from peer ca06c6ed33684aafbaebc0ddd43546f2 for earlier term 2. Current term is 3. Ops: []
I20250110 20:53:25.386322 4288 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 2.48, Last appended by leader: 48, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.386795 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.386933 4288 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:25.387082 3713 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8049f77e2ff44129e129bab33c87b9b in current term 3: Already voted for candidate b286bd11c62645c1a74096177d504b31 in this term.
I20250110 20:53:25.387209 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.387248 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.387270 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 election: Requested vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.387704 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b286bd11c62645c1a74096177d504b31; no voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.387971 4299 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20250110 20:53:25.391927 3779 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.3:37837 timed out after 0.006s (SENT) [suppressed 1 similar messages]
I20250110 20:53:25.392930 3974 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 3.
W20250110 20:53:25.393400 3974 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.2:40555 (request call id 46) took 7 ms (client timeout 6 ms). Trace:
W20250110 20:53:25.393533 3974 rpcz_store.cc:269] 0110 20:53:25.385892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:25.385946 (+ 54us) service_pool.cc:225] Handling call
0110 20:53:25.393273 (+ 7327us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x616000071554. stack: 00007fc21008afd6 00007fc21008ad90 00007fc20f06a48d 000000000096d544 000000000096d50d 000000000096ceda 00007fc21b91eb85 00007fc21f61bce4 00007fc219cee9b6 00007fc219cee6e5 00007fc216df230e 00007fc216df11ee 00007fc216df7425 00007fc216df7d4f 00007fc216df7b78 0000000000a71ecc
0110 20:53:25.393325 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9666304}
I20250110 20:53:25.395171 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.395355 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.396000 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.396121 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 3 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.396396 3974 raft_consensus.cc:2371] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 3. Re-sending same reply.
I20250110 20:53:25.396790 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 4 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.397038 3974 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 3.
I20250110 20:53:25.397341 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31
I20250110 20:53:25.397379 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.397616 4289 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 FOLLOWER]: Leader election won for term 3
I20250110 20:53:25.397665 4299 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250110 20:53:25.397742 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:25.397825 4299 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:53:25.397766 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 4 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.398218 4289 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 LEADER]: Becoming Leader. State: Replica: c8049f77e2ff44129e129bab33c87b9b, State: Running, Role: LEADER
I20250110 20:53:25.398571 4289 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 2.48, Last appended by leader: 48, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.400496 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.401036 3974 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 3 FOLLOWER]: Refusing update from remote peer c8049f77e2ff44129e129bab33c87b9b: Log matching property violated. Preceding OpId in replica: term: 2 index: 48. Preceding OpId from leader: term: 3 index: 49. (index mismatch)
I20250110 20:53:25.401090 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 4 election: Requested vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.401526 4289 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.401305 3570 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b reported cstate change: term changed from 2 to 3, leader changed from ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3) to c8049f77e2ff44129e129bab33c87b9b (127.0.104.2). New cstate: current_term: 3 leader_uuid: "c8049f77e2ff44129e129bab33c87b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:25.402094 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 4 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.402283 3974 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:53:25.402494 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 4 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.404443 3713 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 FOLLOWER]: Rejecting Update request from peer c8049f77e2ff44129e129bab33c87b9b for earlier term 3. Current term is 4. Ops: []
I20250110 20:53:25.404585 3974 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 4.
I20250110 20:53:25.404850 3973 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Rejecting Update request from peer c8049f77e2ff44129e129bab33c87b9b for earlier term 3. Current term is 4. Ops: [3.49-3.49]
I20250110 20:53:25.405449 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2; no voters: c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.405916 4299 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 FOLLOWER]: Leader election won for term 4
I20250110 20:53:25.405896 4335 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: INVALID_TERM, Last received: 2.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.410835 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.411032 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.411902 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.412396 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.412492 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
W20250110 20:53:25.418187 3907 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.006s (SENT) [suppressed 1 similar messages]
I20250110 20:53:25.424116 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.424526 4299 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 LEADER]: Becoming Leader. State: Replica: b286bd11c62645c1a74096177d504b31, State: Running, Role: LEADER
I20250110 20:53:25.425083 3907 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.425040 4299 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 2.48, Last appended by leader: 48, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.425416 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 20:53:25.425671 4289 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 }, Status: INVALID_TERM, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.426414 4338 raft_consensus.cc:3049] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 LEADER]: Stepping down as leader of term 3
W20250110 20:53:25.427294 3713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.3:41559 (request call id 6925) took 15 ms (client timeout 6 ms). Trace:
I20250110 20:53:25.427583 3712 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 LEADER]: Rejecting Update request from peer c8049f77e2ff44129e129bab33c87b9b for earlier term 3. Current term is 4. Ops: [3.49-3.49]
I20250110 20:53:25.427596 3973 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Rejecting Update request from peer c8049f77e2ff44129e129bab33c87b9b for earlier term 3. Current term is 4. Ops: [3.49-3.49]
I20250110 20:53:25.427249 4338 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8049f77e2ff44129e129bab33c87b9b, State: Running, Role: LEADER
W20250110 20:53:25.427435 3713 rpcz_store.cc:269] 0110 20:53:25.412153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:25.412225 (+ 72us) service_pool.cc:225] Handling call
0110 20:53:25.427174 (+ 14949us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x616000071554. stack: 00007ff991a7afd6 00007ff991a7ad90 00007ff990a5a48d 000000000096d544 000000000096d50d 000000000096ceda 00007ff99d30eb85 00007ff9a100bce4 00007ff99b6de9b6 00007ff99b6de6e5 00007ff9987e230e 00007ff9987e11ee 00007ff9987e7425 00007ff9987e7d4f 00007ff9987e7b78 0000000000a71ecc
0110 20:53:25.427238 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33019264}
I20250110 20:53:25.428702 4338 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 3.49, Last appended by leader: 49, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.429107 4289 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: INVALID_TERM, Last received: 2.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.429476 4335 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 }, Status: INVALID_TERM, Last received: 2.48, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.429668 4335 consensus_queue.cc:1225] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [NON_LEADER]: Peer b286bd11c62645c1a74096177d504b31 log is divergent from this leader: its last log entry 4.49 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 48
I20250110 20:53:25.430141 4338 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:53:25.433916 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.434129 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.434456 4338 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.434582 4338 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.434887 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.435318 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.435441 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 2 index: 48 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.436714 3973 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Refusing update from remote peer b286bd11c62645c1a74096177d504b31: Log matching property violated. Preceding OpId in replica: term: 2 index: 48. Preceding OpId from leader: term: 4 index: 49. (index mismatch)
I20250110 20:53:25.437228 4341 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.440254 3843 raft_consensus.cc:2405] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 for term 5 because replica has last-logged OpId of term: 3 index: 49, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 48.
I20250110 20:53:25.440706 3844 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Refusing update from remote peer b286bd11c62645c1a74096177d504b31: Log matching property violated. Preceding OpId in replica: term: 3 index: 49. Preceding OpId from leader: term: 4 index: 49. (term mismatch)
I20250110 20:53:25.440848 3844 pending_rounds.cc:77] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Aborting all ops after (but not including) 48
I20250110 20:53:25.440990 3844 pending_rounds.cc:99] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b: Aborting uncommitted NO_OP operation due to leader change: 3.49
I20250110 20:53:25.440707 4338 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.441092 3844 raft_consensus.cc:2883] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250110 20:53:25.442054 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.442688 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 20:53:25.444460 4342 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20250110 20:53:25.444633 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.444599 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 5 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.445317 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.446272 4338 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
47727b165406405191cd16936164ade5 | 127.0.104.62:33555 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39763 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
b286bd11c62645c1a74096177d504b31 | 127.0.104.1:35003 | HEALTHY | <none> | 0 | 0
c8049f77e2ff44129e129bab33c87b9b | 127.0.104.2:38847 | HEALTHY | <none> | 0 | 0
ca06c6ed33684aafbaebc0ddd43546f2 | 127.0.104.3:37837 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:35003
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:38847
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:37837
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:35003
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:38847
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:37837
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:39763 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 20:53:25.454340 416 log_verifier.cc:126] Checking tablet 26db1d18dd4541db812c55f643f305b3
I20250110 20:53:25.455638 3571 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 reported cstate change: term changed from 3 to 4, leader changed from c8049f77e2ff44129e129bab33c87b9b (127.0.104.2) to b286bd11c62645c1a74096177d504b31 (127.0.104.1). New cstate: current_term: 4 leader_uuid: "b286bd11c62645c1a74096177d504b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:25.466463 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.466670 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.467422 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.468144 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.468320 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.469069 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.469441 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.469496 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.469640 3844 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca06c6ed33684aafbaebc0ddd43546f2 in term 4.
I20250110 20:53:25.470355 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31
I20250110 20:53:25.471573 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.472005 4288 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250110 20:53:25.472110 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.472215 4288 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 4 FOLLOWER]: Advancing to term 5
I20250110 20:53:25.474949 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.475612 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.475942 4335 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250110 20:53:25.476382 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.477185 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.477906 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.477914 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 5 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.478603 3973 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 5 FOLLOWER]: Rejecting Update request from peer b286bd11c62645c1a74096177d504b31 for earlier term 4. Current term is 5. Ops: []
I20250110 20:53:25.478818 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.479349 4298 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20250110 20:53:25.479508 4341 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: INVALID_TERM, Last received: 4.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250110 20:53:25.480413 4299 raft_consensus.cc:3049] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 LEADER]: Stepping down as leader of term 4
I20250110 20:53:25.480515 4299 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b286bd11c62645c1a74096177d504b31, State: Running, Role: LEADER
I20250110 20:53:25.480834 4299 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 4.49, Last appended by leader: 49, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.481158 4299 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 4 FOLLOWER]: Advancing to term 5
I20250110 20:53:25.484722 3973 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 5 FOLLOWER]: Rejecting Update request from peer b286bd11c62645c1a74096177d504b31 for earlier term 4. Current term is 5. Ops: []
I20250110 20:53:25.484869 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.485009 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.485723 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.486344 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 6 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.486574 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 5.
I20250110 20:53:25.486936 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 6 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.487140 3843 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 4.
I20250110 20:53:25.487459 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.487732 4299 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250110 20:53:25.487915 4299 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:25.488065 4299 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 5 FOLLOWER]: Advancing to term 6
I20250110 20:53:25.490821 4299 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.491657 4299 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 6 election: Requested vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.492166 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 6 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.492372 3843 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 4 FOLLOWER]: Advancing to term 6
I20250110 20:53:25.494998 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 6 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.495266 3973 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 5 FOLLOWER]: Advancing to term 6
I20250110 20:53:25.495422 3843 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 6.
W20250110 20:53:25.497968 3648 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.2:38847 timed out after 0.006s (SENT) [suppressed 3 similar messages]
I20250110 20:53:25.498250 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 6.
W20250110 20:53:25.498447 3973 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.1:40829 (request call id 60) took 6 ms (client timeout 6 ms). Trace:
W20250110 20:53:25.498529 3973 rpcz_store.cc:269] 0110 20:53:25.491994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:25.494846 (+ 2852us) service_pool.cc:225] Handling call
0110 20:53:25.498387 (+ 3541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:53:25.499410 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 6 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.499688 3843 raft_consensus.cc:2371] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 6. Re-sending same reply.
I20250110 20:53:25.499817 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 6 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.500015 3973 raft_consensus.cc:2371] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 6. Re-sending same reply.
I20250110 20:53:25.500187 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b; no voters:
I20250110 20:53:25.500499 4299 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 6 FOLLOWER]: Leader election won for term 6
I20250110 20:53:25.500669 4299 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 6 LEADER]: Becoming Leader. State: Replica: b286bd11c62645c1a74096177d504b31, State: Running, Role: LEADER
I20250110 20:53:25.501030 4299 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 4.49, Last appended by leader: 49, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.503021 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.503193 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.503875 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.503760 3572 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "b286bd11c62645c1a74096177d504b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:25.504361 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 7 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.504603 3974 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 6.
I20250110 20:53:25.504704 3843 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Refusing update from remote peer b286bd11c62645c1a74096177d504b31: Log matching property violated. Preceding OpId in replica: term: 4 index: 49. Preceding OpId from leader: term: 6 index: 50. (index mismatch)
I20250110 20:53:25.504936 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 7 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.505424 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.506006 4335 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250110 20:53:25.506106 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:25.506213 4335 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 6 FOLLOWER]: Advancing to term 7
I20250110 20:53:25.506248 4342 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250110 20:53:25.506939 3973 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Refusing update from remote peer b286bd11c62645c1a74096177d504b31: Log matching property violated. Preceding OpId in replica: term: 4 index: 49. Preceding OpId from leader: term: 6 index: 50. (index mismatch)
I20250110 20:53:25.507789 4342 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250110 20:53:25.508832 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.509928 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 7 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.509990 3843 raft_consensus.cc:1235] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Rejecting Update request from peer b286bd11c62645c1a74096177d504b31 for earlier term 6. Current term is 7. Ops: [6.50-6.50]
I20250110 20:53:25.510208 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 7 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.510874 3974 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 7 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.510998 4342 consensus_queue.cc:1046] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 }, Status: INVALID_TERM, Last received: 4.49, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20250110 20:53:25.511333 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.511504 4342 raft_consensus.cc:3049] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 6 LEADER]: Stepping down as leader of term 6
I20250110 20:53:25.511605 4335 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250110 20:53:25.511587 4342 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b286bd11c62645c1a74096177d504b31, State: Running, Role: LEADER
I20250110 20:53:25.512171 4342 consensus_queue.cc:260] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 6.50, Last appended by leader: 50, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.512622 4342 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 6 FOLLOWER]: Advancing to term 7
I20250110 20:53:25.516000 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.516175 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.516762 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.517155 4342 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.517345 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 7 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.517386 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 7 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.517606 3843 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 in current term 7: Already voted for candidate c8049f77e2ff44129e129bab33c87b9b in this term.
I20250110 20:53:25.517377 4342 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.517845 3713 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca06c6ed33684aafbaebc0ddd43546f2 in term 7.
I20250110 20:53:25.518201 3907 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2; no voters: c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.518462 4288 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250110 20:53:25.518553 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.518654 4288 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 6 FOLLOWER]: Advancing to term 7
I20250110 20:53:25.518952 4342 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.519483 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 8 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.519691 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 8 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.519915 3843 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 7.
I20250110 20:53:25.520282 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b; no voters:
I20250110 20:53:25.520604 4342 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250110 20:53:25.520735 4342 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:25.520850 4342 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 7 FOLLOWER]: Advancing to term 8
I20250110 20:53:25.521116 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.521756 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 7 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.521948 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 7.
I20250110 20:53:25.522190 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 7 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.522713 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 7 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.522804 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.522964 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.523355 3843 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 in current term 7: Already voted for candidate c8049f77e2ff44129e129bab33c87b9b in this term.
I20250110 20:53:25.523892 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.524022 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 8 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.524423 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 8 candidate_status { last_received { term: 4 index: 49 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.524574 4342 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.524706 3973 raft_consensus.cc:2405] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8049f77e2ff44129e129bab33c87b9b for term 8 because replica has last-logged OpId of term: 6 index: 50, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 49.
I20250110 20:53:25.524818 416 log_verifier.cc:177] Verified matching terms for 49 ops in tablet 26db1d18dd4541db812c55f643f305b3
I20250110 20:53:25.524878 3713 raft_consensus.cc:2363] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 for earlier term 7. Current term is 8.
I20250110 20:53:25.525542 3907 leader_election.cc:400] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 7 election: Vote denied by peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003) with higher term. Message: Invalid argument: T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 for earlier term 7. Current term is 8.
I20250110 20:53:25.525565 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 8 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.525637 3907 leader_election.cc:403] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20250110 20:53:25.525720 4342 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 8 election: Requested vote from peers c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.525748 3843 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 7 FOLLOWER]: Advancing to term 8
I20250110 20:53:25.526098 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.526871 4288 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 7 FOLLOWER]: Advancing to term 8
I20250110 20:53:25.528242 3843 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 8.
I20250110 20:53:25.528679 3648 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b; no voters:
I20250110 20:53:25.528990 4342 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 FOLLOWER]: Leader election won for term 8
I20250110 20:53:25.529139 4342 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 LEADER]: Becoming Leader. State: Replica: b286bd11c62645c1a74096177d504b31, State: Running, Role: LEADER
I20250110 20:53:25.529300 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003) with higher term. Message: Invalid argument: T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 for earlier term 7. Current term is 8.
I20250110 20:53:25.529464 4342 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 6.50, Last appended by leader: 50, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.529678 4335 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250110 20:53:25.530702 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 8 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.532034 3843 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Refusing update from remote peer b286bd11c62645c1a74096177d504b31: Log matching property violated. Preceding OpId in replica: term: 4 index: 49. Preceding OpId from leader: term: 8 index: 51. (index mismatch)
I20250110 20:53:25.532567 4299 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20250110 20:53:25.532828 3972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "b286bd11c62645c1a74096177d504b31" candidate_term: 8 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.534626 3572 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "b286bd11c62645c1a74096177d504b31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:25.537658 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b286bd11c62645c1a74096177d504b31 in term 8.
I20250110 20:53:25.538021 3974 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Refusing update from remote peer b286bd11c62645c1a74096177d504b31: Log matching property violated. Preceding OpId in replica: term: 6 index: 50. Preceding OpId from leader: term: 8 index: 51. (index mismatch)
W20250110 20:53:25.538515 3973 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.104.1:40829 (request call id 65) took 8 ms (client timeout 6 ms). Trace:
W20250110 20:53:25.538648 3973 rpcz_store.cc:269] 0110 20:53:25.529573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:25.530565 (+ 992us) service_pool.cc:225] Handling call
0110 20:53:25.538481 (+ 7916us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:53:25.539149 4341 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 51, Last known committed idx: 49, Time since last communication: 0.000s
I20250110 20:53:25.539937 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.540109 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.540817 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.541185 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.541868 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 6 index: 50 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.542387 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.542701 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250110 20:53:25.570436 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 3631
I20250110 20:53:25.577807 3753 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:25.583681 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.583775 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.583876 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.583971 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.584518 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.584704 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.585160 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.585237 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.585361 3844 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca06c6ed33684aafbaebc0ddd43546f2 in term 8.
I20250110 20:53:25.585428 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 8.
I20250110 20:53:25.585789 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.585839 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.586081 4335 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250110 20:53:25.586153 4288 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20250110 20:53:25.586198 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Starting leader election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.586241 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b286bd11c62645c1a74096177d504b31)
I20250110 20:53:25.586288 4335 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 8 FOLLOWER]: Advancing to term 9
I20250110 20:53:25.586346 4288 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 8 FOLLOWER]: Advancing to term 9
I20250110 20:53:25.588133 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.588656 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 9 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.589103 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.589330 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.589705 3973 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c8049f77e2ff44129e129bab33c87b9b in current term 9: Already voted for candidate ca06c6ed33684aafbaebc0ddd43546f2 in this term.
I20250110 20:53:25.590003 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.590444 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b"
I20250110 20:53:25.590649 3844 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 in current term 9: Already voted for candidate c8049f77e2ff44129e129bab33c87b9b in this term.
W20250110 20:53:25.597658 3777 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.006s (SENT)
W20250110 20:53:25.600172 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.007s (SENT)
W20250110 20:53:25.606498 3777 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.008s (SENT)
I20250110 20:53:25.606668 3777 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b; no voters: b286bd11c62645c1a74096177d504b31, ca06c6ed33684aafbaebc0ddd43546f2
I20250110 20:53:25.607136 4335 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20250110 20:53:25.607661 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.008s (SENT)
I20250110 20:53:25.607787 3907 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.608093 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20250110 20:53:25.614140 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.614395 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.615157 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.615456 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:25.615633 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.615744 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" is_pre_election: true
I20250110 20:53:25.615957 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 9.
I20250110 20:53:25.616410 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.616499 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:25.616897 4335 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20250110 20:53:25.617017 4335 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:25.616935 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.617125 4335 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 9 FOLLOWER]: Advancing to term 10
I20250110 20:53:25.619912 4335 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.620342 3844 raft_consensus.cc:2388] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ca06c6ed33684aafbaebc0ddd43546f2 in current term 10: Already voted for candidate c8049f77e2ff44129e129bab33c87b9b in this term.
I20250110 20:53:25.620613 4335 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 10 election: Requested vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), ca06c6ed33684aafbaebc0ddd43546f2 (127.0.104.3:37837)
I20250110 20:53:25.621241 3973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "ca06c6ed33684aafbaebc0ddd43546f2"
I20250110 20:53:25.621440 3973 raft_consensus.cc:3054] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 9 FOLLOWER]: Advancing to term 10
I20250110 20:53:25.623317 3973 raft_consensus.cc:2463] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8049f77e2ff44129e129bab33c87b9b in term 10.
I20250110 20:53:25.623736 3779 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c8049f77e2ff44129e129bab33c87b9b, ca06c6ed33684aafbaebc0ddd43546f2; no voters:
I20250110 20:53:25.624027 4335 raft_consensus.cc:2798] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 10 FOLLOWER]: Leader election won for term 10
I20250110 20:53:25.624197 4335 raft_consensus.cc:695] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [term 10 LEADER]: Becoming Leader. State: Replica: c8049f77e2ff44129e129bab33c87b9b, State: Running, Role: LEADER
I20250110 20:53:25.624511 4335 consensus_queue.cc:237] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 51, Committed index: 51, Last appended: 8.51, Last appended by leader: 51, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.627285 3973 raft_consensus.cc:1270] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Refusing update from remote peer c8049f77e2ff44129e129bab33c87b9b: Log matching property violated. Preceding OpId in replica: term: 8 index: 51. Preceding OpId from leader: term: 10 index: 52. (index mismatch)
I20250110 20:53:25.627053 3570 catalog_manager.cc:5526] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b reported cstate change: term changed from 8 to 10, leader changed from b286bd11c62645c1a74096177d504b31 (127.0.104.1) to c8049f77e2ff44129e129bab33c87b9b (127.0.104.2). New cstate: current_term: 10 leader_uuid: "c8049f77e2ff44129e129bab33c87b9b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:25.627919 4335 consensus_queue.cc:1035] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 52, Last known committed idx: 51, Time since last communication: 0.000s
W20250110 20:53:25.643229 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.013s (SENT)
I20250110 20:53:25.643389 3907 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.643661 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20250110 20:53:25.645534 3777 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.013s (SENT)
W20250110 20:53:25.646908 3777 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Timed out: RequestConsensusVote RPC to 127.0.104.1:35003 timed out after 0.012s (SENT)
I20250110 20:53:25.728330 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.729494 3710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.729724 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.729614 3711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.730230 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.730188 3709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.730732 3711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.730636 3705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.730963 3712 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31" is_pre_election: true
I20250110 20:53:25.731036 3713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 10 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.730823 3710 raft_consensus.cc:3049] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 LEADER]: Stepping down as leader of term 8
I20250110 20:53:25.731293 3710 raft_consensus.cc:738] T 26db1d18dd4541db812c55f643f305b3 P b286bd11c62645c1a74096177d504b31 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b286bd11c62645c1a74096177d504b31, State: Running, Role: LEADER
I20250110 20:53:25.731268 3707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "c8049f77e2ff44129e129bab33c87b9b" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "b286bd11c62645c1a74096177d504b31"
I20250110 20:53:25.731473 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 3631
I20250110 20:53:25.757256 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 3762
W20250110 20:53:25.758867 3777 consensus_peers.cc:487] T 26db1d18dd4541db812c55f643f305b3 P c8049f77e2ff44129e129bab33c87b9b -> Peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Couldn't send request to peer b286bd11c62645c1a74096177d504b31. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:35003: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:53:25.763691 3884 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:25.774823 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:53:25.775096 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.775820 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
W20250110 20:53:25.777520 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Network error: Client connection negotiation failed: client connection to 127.0.104.1:35003: connect: Connection refused (error 111)
W20250110 20:53:25.790884 3909 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847): Timed out: RequestConsensusVote RPC to 127.0.104.2:38847 timed out after 0.007s (SENT)
I20250110 20:53:25.791064 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.791381 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250110 20:53:25.799041 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:53:25.799264 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.800029 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
W20250110 20:53:25.801286 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Network error: Client connection negotiation failed: client connection to 127.0.104.1:35003: connect: Connection refused (error 111)
W20250110 20:53:25.827148 3909 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847): Timed out: RequestConsensusVote RPC to 127.0.104.2:38847 timed out after 0.013s (SENT)
I20250110 20:53:25.827332 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.827672 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250110 20:53:25.836396 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:53:25.836606 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.837333 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
W20250110 20:53:25.838533 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Network error: Client connection negotiation failed: client connection to 127.0.104.1:35003: connect: Connection refused (error 111)
W20250110 20:53:25.858536 3909 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847): Timed out: RequestConsensusVote RPC to 127.0.104.2:38847 timed out after 0.010s (SENT)
I20250110 20:53:25.858772 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.859123 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250110 20:53:25.873183 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:53:25.873389 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:25.874032 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
W20250110 20:53:25.875660 3907 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003): Network error: Client connection negotiation failed: client connection to 127.0.104.1:35003: connect: Connection refused (error 111)
I20250110 20:53:25.888870 3843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 11 candidate_status { last_received { term: 10 index: 52 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.888916 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 11 candidate_status { last_received { term: 10 index: 52 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.889441 3842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 11 candidate_status { last_received { term: 10 index: 52 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
I20250110 20:53:25.889987 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 3762
I20250110 20:53:25.889916 3844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26db1d18dd4541db812c55f643f305b3" candidate_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" candidate_term: 11 candidate_status { last_received { term: 10 index: 52 } } ignore_live_leader: false dest_uuid: "c8049f77e2ff44129e129bab33c87b9b" is_pre_election: true
W20250110 20:53:25.905647 3909 leader_election.cc:336] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847): Timed out: RequestConsensusVote RPC to 127.0.104.2:38847 timed out after 0.015s (SENT)
I20250110 20:53:25.905822 3909 leader_election.cc:304] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ca06c6ed33684aafbaebc0ddd43546f2; no voters: b286bd11c62645c1a74096177d504b31, c8049f77e2ff44129e129bab33c87b9b
I20250110 20:53:25.906154 4288 raft_consensus.cc:2743] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20250110 20:53:25.910993 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 3892
W20250110 20:53:25.910979 3909 connection.cc:537] client connection to 127.0.104.2:38847 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 20:53:25.916327 4014 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:26.083243 4288 raft_consensus.cc:491] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c8049f77e2ff44129e129bab33c87b9b)
I20250110 20:53:26.083518 4288 raft_consensus.cc:513] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b286bd11c62645c1a74096177d504b31" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35003 } } peers { permanent_uuid: "c8049f77e2ff44129e129bab33c87b9b" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38847 } } peers { permanent_uuid: "ca06c6ed33684aafbaebc0ddd43546f2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 37837 } }
I20250110 20:53:26.084252 4288 leader_election.cc:290] T 26db1d18dd4541db812c55f643f305b3 P ca06c6ed33684aafbaebc0ddd43546f2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b286bd11c62645c1a74096177d504b31 (127.0.104.1:35003), c8049f77e2ff44129e129bab33c87b9b (127.0.104.2:38847)
I20250110 20:53:26.084723 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 3892
I20250110 20:53:26.113258 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 3541
I20250110 20:53:26.114233 3602 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:26.221848 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 3541
2025-01-10T20:53:26Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (34269 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20250110 20:53:26.260808 416 test_util.cc:274] Using random seed: 1229142050
I20250110 20:53:26.262547 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:53:26.262631 416 ts_itest-base.cc:113] --------------
I20250110 20:53:26.262679 416 ts_itest-base.cc:114] 5 tablet servers
I20250110 20:53:26.262758 416 ts_itest-base.cc:115] 5 replicas per TS
I20250110 20:53:26.262818 416 ts_itest-base.cc:116] --------------
2025-01-10T20:53:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:53:26Z Disabled control of system clock
I20250110 20:53:26.289973 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42769
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42769 with env {}
W20250110 20:53:26.617607 4370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:26.617944 4370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:26.618080 4370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:26.625903 4370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:26.626044 4370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:26.626116 4370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:26.626189 4370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:26.635577 4370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42769
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:26.637176 4370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:26.638950 4370 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:26.649303 4375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:26.649353 4376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:26.649474 4378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:26.649849 4370 server_base.cc:1034] running on GCE node
I20250110 20:53:26.650638 4370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:26.652092 4370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:26.653307 4370 hybrid_clock.cc:648] HybridClock initialized: now 1736542406653232 us; error 51 us; skew 500 ppm
I20250110 20:53:26.653636 4370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:26.655944 4370 webserver.cc:458] Webserver started at http://127.0.104.62:45759/ using document root <none> and password file <none>
I20250110 20:53:26.656687 4370 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:26.656800 4370 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:26.657128 4370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:26.659806 4370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "85367fd4e73f456e9629664973cc4fb8"
format_stamp: "Formatted at 2025-01-10 20:53:26 on dist-test-slave-1h8z"
I20250110 20:53:26.660598 4370 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "85367fd4e73f456e9629664973cc4fb8"
format_stamp: "Formatted at 2025-01-10 20:53:26 on dist-test-slave-1h8z"
I20250110 20:53:26.665935 4370 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 20:53:26.669544 4384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:26.671017 4370 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:26.671193 4370 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "85367fd4e73f456e9629664973cc4fb8"
format_stamp: "Formatted at 2025-01-10 20:53:26 on dist-test-slave-1h8z"
I20250110 20:53:26.671387 4370 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:26.697521 4370 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:26.698413 4370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:26.698694 4370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:26.715945 4370 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42769
I20250110 20:53:26.715973 4435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42769 every 8 connection(s)
I20250110 20:53:26.717450 4370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:26.718879 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 4370
I20250110 20:53:26.719086 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:53:26.722460 4436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:26.733043 4436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8: Bootstrap starting.
I20250110 20:53:26.736215 4436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:26.737389 4436 log.cc:826] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:26.740008 4436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8: No bootstrap required, opened a new log
I20250110 20:53:26.746191 4436 raft_consensus.cc:357] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85367fd4e73f456e9629664973cc4fb8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42769 } }
I20250110 20:53:26.746591 4436 raft_consensus.cc:383] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:26.746706 4436 raft_consensus.cc:738] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85367fd4e73f456e9629664973cc4fb8, State: Initialized, Role: FOLLOWER
I20250110 20:53:26.747378 4436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85367fd4e73f456e9629664973cc4fb8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42769 } }
I20250110 20:53:26.747576 4436 raft_consensus.cc:397] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:26.747697 4436 raft_consensus.cc:491] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:26.747840 4436 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:26.749845 4436 raft_consensus.cc:513] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85367fd4e73f456e9629664973cc4fb8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42769 } }
I20250110 20:53:26.750355 4436 leader_election.cc:304] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85367fd4e73f456e9629664973cc4fb8; no voters:
I20250110 20:53:26.750924 4436 leader_election.cc:290] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:26.751062 4441 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:26.751631 4441 raft_consensus.cc:695] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [term 1 LEADER]: Becoming Leader. State: Replica: 85367fd4e73f456e9629664973cc4fb8, State: Running, Role: LEADER
I20250110 20:53:26.752142 4441 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85367fd4e73f456e9629664973cc4fb8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42769 } }
I20250110 20:53:26.752980 4436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:26.754912 4443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85367fd4e73f456e9629664973cc4fb8. Latest consensus state: current_term: 1 leader_uuid: "85367fd4e73f456e9629664973cc4fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85367fd4e73f456e9629664973cc4fb8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42769 } } }
I20250110 20:53:26.755482 4443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:26.755404 4442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85367fd4e73f456e9629664973cc4fb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85367fd4e73f456e9629664973cc4fb8" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42769 } } }
I20250110 20:53:26.755667 4442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:26.760119 4450 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:26.764780 4450 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:26.773308 4450 catalog_manager.cc:1348] Generated new cluster ID: a0f84c97b18343bcaaf072a59c710f69
I20250110 20:53:26.773428 4450 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:26.787438 4450 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:53:26.788645 4450 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:26.797111 4450 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 85367fd4e73f456e9629664973cc4fb8: Generated new TSK 0
I20250110 20:53:26.797935 4450 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:53:26.804641 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20250110 20:53:27.135561 4460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:27.135862 4460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:27.135964 4460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:27.143762 4460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:27.143991 4460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:27.153358 4460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:27.154997 4460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:27.156672 4460 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:27.168363 4465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:27.168394 4466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:27.168373 4468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:27.169340 4460 server_base.cc:1034] running on GCE node
I20250110 20:53:27.169909 4460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:27.170917 4460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:27.172158 4460 hybrid_clock.cc:648] HybridClock initialized: now 1736542407172082 us; error 59 us; skew 500 ppm
I20250110 20:53:27.172482 4460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:27.174785 4460 webserver.cc:458] Webserver started at http://127.0.104.1:40979/ using document root <none> and password file <none>
I20250110 20:53:27.175597 4460 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:27.175740 4460 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:27.176049 4460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:27.178891 4460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fed858ce3e3d428ea64e46e861c8d5b7"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:27.179661 4460 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fed858ce3e3d428ea64e46e861c8d5b7"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:27.184974 4460 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:53:27.188397 4474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:27.189771 4460 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250110 20:53:27.189939 4460 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fed858ce3e3d428ea64e46e861c8d5b7"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:27.190122 4460 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:27.207975 4460 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:27.208843 4460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:27.209115 4460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:27.210079 4460 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:27.211639 4460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:27.211719 4460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:27.211840 4460 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:27.211892 4460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:27.241334 4460 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36597
I20250110 20:53:27.241374 4586 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36597 every 8 connection(s)
I20250110 20:53:27.242611 4460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:27.244239 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 4460
I20250110 20:53:27.244407 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:53:27.247843 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250110 20:53:27.256094 4587 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:27.256454 4587 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:27.257486 4587 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:27.259721 4401 ts_manager.cc:194] Registered new tserver with Master: fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597)
I20250110 20:53:27.261328 4401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:47617
W20250110 20:53:27.574980 4591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:27.575286 4591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:27.575413 4591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:27.583295 4591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:27.583527 4591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:53:27.592857 4591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:27.594432 4591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:27.596272 4591 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:27.607527 4599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:27.607513 4597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:27.607525 4596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:27.608552 4591 server_base.cc:1034] running on GCE node
I20250110 20:53:27.609169 4591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:27.610090 4591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:27.611317 4591 hybrid_clock.cc:648] HybridClock initialized: now 1736542407611232 us; error 42 us; skew 500 ppm
I20250110 20:53:27.611656 4591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:27.614446 4591 webserver.cc:458] Webserver started at http://127.0.104.2:35453/ using document root <none> and password file <none>
I20250110 20:53:27.615392 4591 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:27.615514 4591 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:27.615800 4591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:27.618427 4591 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9902c01dcf24446aa6c1f6c17aba77d0"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:27.619223 4591 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9902c01dcf24446aa6c1f6c17aba77d0"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:27.624421 4591 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250110 20:53:27.627974 4605 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:27.629364 4591 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250110 20:53:27.629575 4591 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9902c01dcf24446aa6c1f6c17aba77d0"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:27.629765 4591 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:27.661901 4591 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:27.662801 4591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:27.663069 4591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:27.664034 4591 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:27.665583 4591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:27.665660 4591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:27.665782 4591 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:27.665833 4591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:27.694969 4591 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:36031
I20250110 20:53:27.695027 4717 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:36031 every 8 connection(s)
I20250110 20:53:27.696369 4591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:53:27.698391 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 4591
I20250110 20:53:27.698596 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:53:27.701824 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250110 20:53:27.709776 4718 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:27.710100 4718 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:27.710974 4718 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:27.712425 4401 ts_manager.cc:194] Registered new tserver with Master: 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031)
I20250110 20:53:27.713407 4401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:44307
W20250110 20:53:28.027652 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:28.027951 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:28.028038 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:28.035974 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:28.036198 4721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:53:28.045500 4721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:28.047097 4721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:28.048743 4721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:28.059881 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:28.059916 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:28.059937 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:28.060827 4721 server_base.cc:1034] running on GCE node
I20250110 20:53:28.061414 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:28.062362 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:28.063568 4721 hybrid_clock.cc:648] HybridClock initialized: now 1736542408063497 us; error 47 us; skew 500 ppm
I20250110 20:53:28.063861 4721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:28.066118 4721 webserver.cc:458] Webserver started at http://127.0.104.3:42437/ using document root <none> and password file <none>
I20250110 20:53:28.066895 4721 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:28.067021 4721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:28.067325 4721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:28.069901 4721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5137413ae3884ca68ee8f187db732e8e"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.070654 4721 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5137413ae3884ca68ee8f187db732e8e"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.075860 4721 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250110 20:53:28.079351 4735 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.080724 4721 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250110 20:53:28.080876 4721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5137413ae3884ca68ee8f187db732e8e"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.081074 4721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:28.100971 4721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:28.101805 4721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:28.102078 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:28.103082 4721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:28.104796 4721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:28.104877 4721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.104976 4721 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:28.105017 4721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.133874 4721 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44051
I20250110 20:53:28.133910 4847 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44051 every 8 connection(s)
I20250110 20:53:28.135319 4721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:53:28.143376 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 4721
I20250110 20:53:28.143556 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:53:28.146565 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.4:0
--local_ip_for_outbound_sockets=127.0.104.4
--webserver_interface=127.0.104.4
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250110 20:53:28.147722 4848 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:28.148077 4848 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:28.149068 4848 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:28.150635 4401 ts_manager.cc:194] Registered new tserver with Master: 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051)
I20250110 20:53:28.151613 4401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:34737
I20250110 20:53:28.264868 4587 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
W20250110 20:53:28.467376 4851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:28.467662 4851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:28.467773 4851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:28.475548 4851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:28.475768 4851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.4
I20250110 20:53:28.485129 4851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.104.4
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.4
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:28.486832 4851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:28.488513 4851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:28.499477 4856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:28.499537 4857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:28.499557 4859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:28.500496 4851 server_base.cc:1034] running on GCE node
I20250110 20:53:28.501066 4851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:28.501963 4851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:28.503175 4851 hybrid_clock.cc:648] HybridClock initialized: now 1736542408503105 us; error 41 us; skew 500 ppm
I20250110 20:53:28.503471 4851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:28.505745 4851 webserver.cc:458] Webserver started at http://127.0.104.4:35211/ using document root <none> and password file <none>
I20250110 20:53:28.506471 4851 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:28.506585 4851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:28.506904 4851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:28.509479 4851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "9860e3170fbc43c0b7d4fc77e60c3490"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.510236 4851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "9860e3170fbc43c0b7d4fc77e60c3490"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.515537 4851 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20250110 20:53:28.519032 4865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.520406 4851 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250110 20:53:28.520570 4851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9860e3170fbc43c0b7d4fc77e60c3490"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.520742 4851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:28.536739 4851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:28.537590 4851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:28.537849 4851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:28.538766 4851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:28.540269 4851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:28.540343 4851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.540462 4851 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:28.540515 4851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.569162 4851 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.4:36075
I20250110 20:53:28.569216 4977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.4:36075 every 8 connection(s)
I20250110 20:53:28.570401 4851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 20:53:28.577049 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 4851
I20250110 20:53:28.577261 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 20:53:28.580724 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.5:0
--local_ip_for_outbound_sockets=127.0.104.5
--webserver_interface=127.0.104.5
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250110 20:53:28.584168 4978 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:28.584515 4978 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:28.585567 4978 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:28.587352 4401 ts_manager.cc:194] Registered new tserver with Master: 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075)
I20250110 20:53:28.588460 4401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.4:55325
I20250110 20:53:28.716202 4718 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
W20250110 20:53:28.908397 4981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:28.908694 4981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:28.908814 4981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:28.916503 4981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:28.916738 4981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.5
I20250110 20:53:28.926209 4981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.104.5
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.5
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:28.928297 4981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:28.930586 4981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:28.942245 4987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:28.942195 4989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:28.942209 4986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:28.942955 4981 server_base.cc:1034] running on GCE node
I20250110 20:53:28.943562 4981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:28.944530 4981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:28.945741 4981 hybrid_clock.cc:648] HybridClock initialized: now 1736542408945675 us; error 47 us; skew 500 ppm
I20250110 20:53:28.946044 4981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:28.948215 4981 webserver.cc:458] Webserver started at http://127.0.104.5:32991/ using document root <none> and password file <none>
I20250110 20:53:28.948989 4981 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:28.949105 4981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:28.949393 4981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:28.952008 4981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "5d21cd10ee8f4606aee63432a45ec7c3"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.952741 4981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "5d21cd10ee8f4606aee63432a45ec7c3"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.958034 4981 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 20:53:28.961540 4995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.963133 4981 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:28.963402 4981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "5d21cd10ee8f4606aee63432a45ec7c3"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:28.963613 4981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:28.981168 4981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:28.982030 4981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:28.982306 4981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:28.983278 4981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:28.984778 4981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:28.984855 4981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:28.984973 4981 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:28.985029 4981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:29.013916 4981 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.5:42353
I20250110 20:53:29.013962 5107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.5:42353 every 8 connection(s)
I20250110 20:53:29.015239 4981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 20:53:29.022015 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 4981
I20250110 20:53:29.022197 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 20:53:29.027761 5108 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:29.028043 5108 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:29.028854 5108 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:29.030218 4401 ts_manager.cc:194] Registered new tserver with Master: 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:29.031049 4401 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.5:56301
I20250110 20:53:29.037842 416 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 20:53:29.052902 4401 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42282:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250110 20:53:29.058494 4401 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20250110 20:53:29.103356 4653 tablet_service.cc:1467] Processing CreateTablet for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3 (DEFAULT_TABLE table=TestTable [id=737bc23f17f740a0852bd193151b0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:29.103356 4913 tablet_service.cc:1467] Processing CreateTablet for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3 (DEFAULT_TABLE table=TestTable [id=737bc23f17f740a0852bd193151b0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:29.104875 4913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:29.106415 4653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:29.109838 4522 tablet_service.cc:1467] Processing CreateTablet for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3 (DEFAULT_TABLE table=TestTable [id=737bc23f17f740a0852bd193151b0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:29.113709 5043 tablet_service.cc:1467] Processing CreateTablet for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3 (DEFAULT_TABLE table=TestTable [id=737bc23f17f740a0852bd193151b0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:29.115276 5043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:29.123602 4522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:29.131759 4783 tablet_service.cc:1467] Processing CreateTablet for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3 (DEFAULT_TABLE table=TestTable [id=737bc23f17f740a0852bd193151b0ce1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:29.133529 4783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58912c24a6f040dcb3ab3ff65a0c1cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:29.145820 5129 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: Bootstrap starting.
I20250110 20:53:29.148229 5130 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Bootstrap starting.
I20250110 20:53:29.149327 5131 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: Bootstrap starting.
I20250110 20:53:29.150208 5129 tablet_bootstrap.cc:654] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:29.151571 5129 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:29.152073 5130 tablet_bootstrap.cc:654] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:29.152685 5131 tablet_bootstrap.cc:654] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:29.153882 5131 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:29.154136 5130 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:29.154556 4848 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
I20250110 20:53:29.156736 5132 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Bootstrap starting.
I20250110 20:53:29.160734 5132 tablet_bootstrap.cc:654] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:29.162531 5134 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: Bootstrap starting.
I20250110 20:53:29.163339 5132 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:29.166514 5134 tablet_bootstrap.cc:654] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:29.166541 5129 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: No bootstrap required, opened a new log
I20250110 20:53:29.166903 5129 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.004s
I20250110 20:53:29.167945 5134 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:29.170703 5131 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: No bootstrap required, opened a new log
I20250110 20:53:29.171065 5131 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.008s
I20250110 20:53:29.173148 5130 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: No bootstrap required, opened a new log
I20250110 20:53:29.173509 5130 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.012s
I20250110 20:53:29.179015 5129 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.179551 5129 raft_consensus.cc:383] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:29.179651 5129 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9902c01dcf24446aa6c1f6c17aba77d0, State: Initialized, Role: FOLLOWER
I20250110 20:53:29.180511 5129 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.180974 5131 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.181505 5131 raft_consensus.cc:383] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:29.181612 5131 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d21cd10ee8f4606aee63432a45ec7c3, State: Initialized, Role: FOLLOWER
I20250110 20:53:29.182467 5131 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.183964 5130 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.184468 5130 raft_consensus.cc:383] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:29.184571 5130 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9860e3170fbc43c0b7d4fc77e60c3490, State: Initialized, Role: FOLLOWER
I20250110 20:53:29.185331 5130 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.185874 5132 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: No bootstrap required, opened a new log
I20250110 20:53:29.186262 5132 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.016s
I20250110 20:53:29.189356 5129 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: Time spent starting tablet: real 0.022s user 0.014s sys 0.004s
I20250110 20:53:29.191824 5134 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: No bootstrap required, opened a new log
I20250110 20:53:29.192194 5134 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.000s
I20250110 20:53:29.196455 4978 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
I20250110 20:53:29.196448 5108 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
I20250110 20:53:29.196727 5130 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Time spent starting tablet: real 0.023s user 0.014s sys 0.006s
I20250110 20:53:29.196934 5131 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: Time spent starting tablet: real 0.026s user 0.012s sys 0.012s
W20250110 20:53:29.198881 4719 tablet.cc:2367] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:29.201887 5134 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.202423 5134 raft_consensus.cc:383] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:29.202529 5134 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5137413ae3884ca68ee8f187db732e8e, State: Initialized, Role: FOLLOWER
I20250110 20:53:29.203301 5134 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.203859 5132 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.204334 5132 raft_consensus.cc:383] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:29.204439 5132 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed858ce3e3d428ea64e46e861c8d5b7, State: Initialized, Role: FOLLOWER
I20250110 20:53:29.205113 5134 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20250110 20:53:29.205253 5132 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.206955 5132 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Time spent starting tablet: real 0.021s user 0.012s sys 0.000s
W20250110 20:53:29.245575 4588 tablet.cc:2367] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:29.266817 5109 tablet.cc:2367] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:29.322417 4979 tablet.cc:2367] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:29.387494 4849 tablet.cc:2367] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:29.396476 5143 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:29.396742 5143 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.398697 5143 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:29.408614 4933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" is_pre_election: true
I20250110 20:53:29.408602 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" is_pre_election: true
I20250110 20:53:29.409196 5063 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 0.
I20250110 20:53:29.409201 4933 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 0.
I20250110 20:53:29.410010 4476 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5d21cd10ee8f4606aee63432a45ec7c3, 9860e3170fbc43c0b7d4fc77e60c3490, fed858ce3e3d428ea64e46e861c8d5b7; no voters:
I20250110 20:53:29.410631 5143 raft_consensus.cc:2798] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:53:29.410830 5143 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:29.410925 5143 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:29.414324 5143 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.415408 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" is_pre_election: true
I20250110 20:53:29.415938 4673 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 0.
I20250110 20:53:29.416203 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3"
I20250110 20:53:29.416425 5063 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:29.416807 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0"
I20250110 20:53:29.417006 4673 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:29.419685 5063 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 1.
I20250110 20:53:29.419793 4673 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 1.
I20250110 20:53:29.419589 4803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e" is_pre_election: true
I20250110 20:53:29.420185 4803 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 0.
I20250110 20:53:29.420711 4478 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5d21cd10ee8f4606aee63432a45ec7c3, 9902c01dcf24446aa6c1f6c17aba77d0, fed858ce3e3d428ea64e46e861c8d5b7; no voters:
I20250110 20:53:29.420635 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e"
I20250110 20:53:29.420797 5143 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [CANDIDATE]: Term 1 election: Requested vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:29.420848 4933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9860e3170fbc43c0b7d4fc77e60c3490"
I20250110 20:53:29.420917 4802 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:29.421047 4933 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:29.421206 5143 raft_consensus.cc:2798] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:29.421355 5143 raft_consensus.cc:695] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 1 LEADER]: Becoming Leader. State: Replica: fed858ce3e3d428ea64e46e861c8d5b7, State: Running, Role: LEADER
I20250110 20:53:29.421893 5143 consensus_queue.cc:237] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:29.423786 4802 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 1.
I20250110 20:53:29.424269 4933 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fed858ce3e3d428ea64e46e861c8d5b7 in term 1.
I20250110 20:53:29.427863 4401 catalog_manager.cc:5526] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:29.447283 416 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 20:53:29.449384 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver fed858ce3e3d428ea64e46e861c8d5b7 to finish bootstrapping
I20250110 20:53:29.458250 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9902c01dcf24446aa6c1f6c17aba77d0 to finish bootstrapping
I20250110 20:53:29.466470 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5137413ae3884ca68ee8f187db732e8e to finish bootstrapping
I20250110 20:53:29.474143 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9860e3170fbc43c0b7d4fc77e60c3490 to finish bootstrapping
I20250110 20:53:29.481091 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5d21cd10ee8f4606aee63432a45ec7c3 to finish bootstrapping
I20250110 20:53:29.496917 416 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
I20250110 20:53:29.518692 4933 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Refusing update from remote peer fed858ce3e3d428ea64e46e861c8d5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:29.518730 5063 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Refusing update from remote peer fed858ce3e3d428ea64e46e861c8d5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:29.518730 4802 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 1 FOLLOWER]: Refusing update from remote peer fed858ce3e3d428ea64e46e861c8d5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:29.519783 5153 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:29.520107 5154 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:29.520190 4673 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Refusing update from remote peer fed858ce3e3d428ea64e46e861c8d5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:29.520447 5143 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:29.520915 5154 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:29.552310 5155 mvcc.cc:204] Tried to move back new op lower bound from 7112877709377441792 to 7112877708998717440. Current Snapshot: MvccSnapshot[applied={T|T < 7112877709377441792}]
I20250110 20:53:29.552747 5157 mvcc.cc:204] Tried to move back new op lower bound from 7112877709377441792 to 7112877708998717440. Current Snapshot: MvccSnapshot[applied={T|T < 7112877709377441792}]
I20250110 20:53:29.564790 5158 mvcc.cc:204] Tried to move back new op lower bound from 7112877709377441792 to 7112877708998717440. Current Snapshot: MvccSnapshot[applied={T|T < 7112877709377441792 or (T in {7112877709377441792})}]
I20250110 20:53:29.603924 416 raft_consensus_election-itest.cc:408] Killing current leader fed858ce3e3d428ea64e46e861c8d5b7...
I20250110 20:53:29.604094 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 4460
I20250110 20:53:29.605193 4582 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:29.696210 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 4460
I20250110 20:53:29.718480 416 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20250110 20:53:29.721284 416 test_util.cc:274] Using random seed: 1232602522
I20250110 20:53:29.849838 416 test_util.cc:274] Using random seed: 1232731091
I20250110 20:53:30.057919 416 test_util.cc:274] Using random seed: 1232939144
I20250110 20:53:30.365556 416 test_util.cc:274] Using random seed: 1233246806
I20250110 20:53:30.773284 416 test_util.cc:274] Using random seed: 1233654514
I20250110 20:53:31.103967 5170 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fed858ce3e3d428ea64e46e861c8d5b7)
I20250110 20:53:31.104076 5171 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fed858ce3e3d428ea64e46e861c8d5b7)
I20250110 20:53:31.104192 5170 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:31.104532 5172 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fed858ce3e3d428ea64e46e861c8d5b7)
I20250110 20:53:31.104804 5173 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fed858ce3e3d428ea64e46e861c8d5b7)
I20250110 20:53:31.104758 5172 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:31.104794 5171 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:31.105010 5173 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:31.106513 5170 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:31.112861 5171 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:31.120190 5172 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
W20250110 20:53:31.140904 4869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.150697 4932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" is_pre_election: true
I20250110 20:53:31.151006 4932 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5137413ae3884ca68ee8f187db732e8e in term 1.
I20250110 20:53:31.159693 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e" is_pre_election: true
I20250110 20:53:31.159977 4802 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in term 1.
W20250110 20:53:31.161495 4869 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.171176 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" is_pre_election: true
I20250110 20:53:31.171463 4673 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in term 1.
I20250110 20:53:31.171908 5173 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051)
I20250110 20:53:31.172701 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" is_pre_election: true
I20250110 20:53:31.172902 4673 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5137413ae3884ca68ee8f187db732e8e in term 1.
W20250110 20:53:31.173064 4739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.173130 4869 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 5137413ae3884ca68ee8f187db732e8e, 9860e3170fbc43c0b7d4fc77e60c3490, 9902c01dcf24446aa6c1f6c17aba77d0; no voters: fed858ce3e3d428ea64e46e861c8d5b7
I20250110 20:53:31.173564 5171 raft_consensus.cc:2798] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:53:31.173692 5171 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fed858ce3e3d428ea64e46e861c8d5b7)
W20250110 20:53:31.173666 4609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.173792 5171 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:31.176959 5171 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:31.177902 5171 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [CANDIDATE]: Term 2 election: Requested vote from peers 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:31.179569 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e"
I20250110 20:53:31.179785 4802 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 1 FOLLOWER]: Advancing to term 2
W20250110 20:53:31.180860 4869 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.182180 4932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" is_pre_election: true
I20250110 20:53:31.182444 4932 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d21cd10ee8f4606aee63432a45ec7c3 in current term 2: Already voted for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in this term.
I20250110 20:53:31.182608 4802 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in term 2.
I20250110 20:53:31.172516 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" is_pre_election: true
I20250110 20:53:31.183014 4932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" is_pre_election: true
I20250110 20:53:31.183054 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0"
I20250110 20:53:31.183223 4932 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9902c01dcf24446aa6c1f6c17aba77d0 in current term 2: Already voted for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in this term.
I20250110 20:53:31.183185 5063 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9902c01dcf24446aa6c1f6c17aba77d0 in term 1.
I20250110 20:53:31.183238 4673 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:31.178303 5062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" is_pre_election: true
I20250110 20:53:31.183638 5062 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in term 1.
I20250110 20:53:31.183916 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3"
I20250110 20:53:31.184110 5063 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:31.186237 4673 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in term 2.
I20250110 20:53:31.186725 4869 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 5137413ae3884ca68ee8f187db732e8e, 9860e3170fbc43c0b7d4fc77e60c3490, 9902c01dcf24446aa6c1f6c17aba77d0; no voters: fed858ce3e3d428ea64e46e861c8d5b7
I20250110 20:53:31.186935 5063 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in term 2.
I20250110 20:53:31.187224 5171 raft_consensus.cc:2798] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:53:31.190285 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e" is_pre_election: true
W20250110 20:53:31.190481 4609 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.190783 4802 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9902c01dcf24446aa6c1f6c17aba77d0 in current term 2: Already voted for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in this term.
I20250110 20:53:31.191403 4609 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 5d21cd10ee8f4606aee63432a45ec7c3, 9902c01dcf24446aa6c1f6c17aba77d0; no voters: 5137413ae3884ca68ee8f187db732e8e, 9860e3170fbc43c0b7d4fc77e60c3490, fed858ce3e3d428ea64e46e861c8d5b7
I20250110 20:53:31.192283 5170 raft_consensus.cc:2743] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:53:31.193473 4999 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.194762 4739 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5137413ae3884ca68ee8f187db732e8e, 9860e3170fbc43c0b7d4fc77e60c3490, 9902c01dcf24446aa6c1f6c17aba77d0; no voters:
I20250110 20:53:31.195243 5172 raft_consensus.cc:2758] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
W20250110 20:53:31.195626 4739 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.197042 5171 raft_consensus.cc:695] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 LEADER]: Becoming Leader. State: Replica: 9860e3170fbc43c0b7d4fc77e60c3490, State: Running, Role: LEADER
I20250110 20:53:31.197971 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" is_pre_election: true
I20250110 20:53:31.198334 5063 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5137413ae3884ca68ee8f187db732e8e in current term 2: Already voted for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in this term.
I20250110 20:53:31.198279 4673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" is_pre_election: true
I20250110 20:53:31.198534 4673 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d21cd10ee8f4606aee63432a45ec7c3 in current term 2: Already voted for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in this term.
W20250110 20:53:31.198596 4999 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.197700 5171 consensus_queue.cc:237] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:31.199826 4999 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 5d21cd10ee8f4606aee63432a45ec7c3; no voters: 9860e3170fbc43c0b7d4fc77e60c3490, 9902c01dcf24446aa6c1f6c17aba77d0, fed858ce3e3d428ea64e46e861c8d5b7
I20250110 20:53:31.200271 5173 raft_consensus.cc:2743] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:53:31.200627 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e" is_pre_election: true
I20250110 20:53:31.200860 4802 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d21cd10ee8f4606aee63432a45ec7c3 in current term 2: Already voted for candidate 9860e3170fbc43c0b7d4fc77e60c3490 in this term.
I20250110 20:53:31.202796 4399 catalog_manager.cc:5526] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 reported cstate change: term changed from 1 to 2, leader changed from fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1) to 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4). New cstate: current_term: 2 leader_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:31.281749 416 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20250110 20:53:31.285296 5114 meta_cache.cc:302] tablet 58912c24a6f040dcb3ab3ff65a0c1cd3: replica fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597) has failed: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:31.299412 5063 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Refusing update from remote peer 9860e3170fbc43c0b7d4fc77e60c3490: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 20:53:31.299412 4673 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Refusing update from remote peer 9860e3170fbc43c0b7d4fc77e60c3490: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 20:53:31.299414 4802 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Refusing update from remote peer 9860e3170fbc43c0b7d4fc77e60c3490: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250110 20:53:31.300150 5197 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 20:53:31.300482 5207 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 20:53:31.300730 5171 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 20:53:31.300944 4869 consensus_peers.cc:487] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 -> Peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Couldn't send request to peer fed858ce3e3d428ea64e46e861c8d5b7. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:53:31.317866 5209 mvcc.cc:204] Tried to move back new op lower bound from 7112877716674863104 to 7112877716275716096. Current Snapshot: MvccSnapshot[applied={T|T < 7112877716674863104}]
I20250110 20:53:31.353018 416 raft_consensus_election-itest.cc:408] Killing current leader 9860e3170fbc43c0b7d4fc77e60c3490...
I20250110 20:53:31.353176 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 4851
I20250110 20:53:31.354187 4973 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:31.445689 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 4851
I20250110 20:53:31.460176 416 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20250110 20:53:31.463140 416 test_util.cc:274] Using random seed: 1234344366
I20250110 20:53:31.570166 416 test_util.cc:274] Using random seed: 1234451412
I20250110 20:53:31.777729 416 test_util.cc:274] Using random seed: 1234658941
I20250110 20:53:32.085027 416 test_util.cc:274] Using random seed: 1234966276
I20250110 20:53:32.492417 416 test_util.cc:274] Using random seed: 1235373648
I20250110 20:53:32.852993 5222 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9860e3170fbc43c0b7d4fc77e60c3490)
I20250110 20:53:32.853190 5222 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:32.853698 5221 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9860e3170fbc43c0b7d4fc77e60c3490)
I20250110 20:53:32.853863 5223 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9860e3170fbc43c0b7d4fc77e60c3490)
I20250110 20:53:32.853873 5221 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:32.854041 5223 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:32.855582 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" is_pre_election: true
I20250110 20:53:32.855556 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" is_pre_election: true
I20250110 20:53:32.855777 4672 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5137413ae3884ca68ee8f187db732e8e in term 2.
I20250110 20:53:32.855784 5063 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5137413ae3884ca68ee8f187db732e8e in term 2.
I20250110 20:53:32.856320 5222 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:32.856830 4739 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 5137413ae3884ca68ee8f187db732e8e, 5d21cd10ee8f4606aee63432a45ec7c3, 9902c01dcf24446aa6c1f6c17aba77d0; no voters:
I20250110 20:53:32.857395 5222 raft_consensus.cc:2798] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:53:32.857550 5222 raft_consensus.cc:491] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9860e3170fbc43c0b7d4fc77e60c3490)
I20250110 20:53:32.857684 5222 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:32.858275 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" is_pre_election: true
I20250110 20:53:32.858500 4672 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d21cd10ee8f4606aee63432a45ec7c3 in term 2.
I20250110 20:53:32.860617 5222 raft_consensus.cc:513] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:32.865381 5223 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051)
I20250110 20:53:32.866469 5222 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 election: Requested vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), 9902c01dcf24446aa6c1f6c17aba77d0 (127.0.104.2:36031), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
I20250110 20:53:32.866899 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3"
I20250110 20:53:32.867136 5063 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:32.870244 5063 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5137413ae3884ca68ee8f187db732e8e in term 3.
I20250110 20:53:32.871610 5221 leader_election.cc:290] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075), fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597), 5137413ae3884ca68ee8f187db732e8e (127.0.104.3:44051), 5d21cd10ee8f4606aee63432a45ec7c3 (127.0.104.5:42353)
W20250110 20:53:32.872162 4737 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075): Network error: Client connection negotiation failed: client connection to 127.0.104.4:36075: connect: Connection refused (error 111)
W20250110 20:53:32.872184 4999 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:32.872539 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e" is_pre_election: true
I20250110 20:53:32.872788 4802 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9902c01dcf24446aa6c1f6c17aba77d0 in current term 3: Already voted for candidate 5137413ae3884ca68ee8f187db732e8e in this term.
I20250110 20:53:32.872987 4672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5137413ae3884ca68ee8f187db732e8e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "9902c01dcf24446aa6c1f6c17aba77d0"
I20250110 20:53:32.873193 4672 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:32.873167 5063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" is_pre_election: true
I20250110 20:53:32.873412 5063 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9902c01dcf24446aa6c1f6c17aba77d0 in current term 3: Already voted for candidate 5137413ae3884ca68ee8f187db732e8e in this term.
W20250110 20:53:32.873644 4997 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075): Network error: Client connection negotiation failed: client connection to 127.0.104.4:36075: connect: Connection refused (error 111)
I20250110 20:53:32.874771 4802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58912c24a6f040dcb3ab3ff65a0c1cd3" candidate_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "5137413ae3884ca68ee8f187db732e8e" is_pre_election: true
W20250110 20:53:32.874889 4607 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075): Network error: Client connection negotiation failed: client connection to 127.0.104.4:36075: connect: Connection refused (error 111)
I20250110 20:53:32.875016 4802 raft_consensus.cc:2388] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5d21cd10ee8f4606aee63432a45ec7c3 in current term 3: Already voted for candidate 5137413ae3884ca68ee8f187db732e8e in this term.
I20250110 20:53:32.875052 4607 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9902c01dcf24446aa6c1f6c17aba77d0; no voters: 5137413ae3884ca68ee8f187db732e8e, 5d21cd10ee8f4606aee63432a45ec7c3, 9860e3170fbc43c0b7d4fc77e60c3490
W20250110 20:53:32.875051 4739 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
W20250110 20:53:32.875326 4737 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075): Network error: Client connection negotiation failed: client connection to 127.0.104.4:36075: connect: Connection refused (error 111)
W20250110 20:53:32.875499 4609 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:32.875485 4999 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 5d21cd10ee8f4606aee63432a45ec7c3, 9902c01dcf24446aa6c1f6c17aba77d0; no voters: 5137413ae3884ca68ee8f187db732e8e, 9860e3170fbc43c0b7d4fc77e60c3490, fed858ce3e3d428ea64e46e861c8d5b7
I20250110 20:53:32.875768 5223 raft_consensus.cc:2743] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250110 20:53:32.876451 4739 leader_election.cc:336] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111)
I20250110 20:53:32.876876 4672 raft_consensus.cc:2463] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5137413ae3884ca68ee8f187db732e8e in term 3.
I20250110 20:53:32.877048 5221 raft_consensus.cc:2743] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:53:32.877259 4739 leader_election.cc:304] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 5137413ae3884ca68ee8f187db732e8e, 5d21cd10ee8f4606aee63432a45ec7c3, 9902c01dcf24446aa6c1f6c17aba77d0; no voters: 9860e3170fbc43c0b7d4fc77e60c3490, fed858ce3e3d428ea64e46e861c8d5b7
I20250110 20:53:32.877543 5222 raft_consensus.cc:2798] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 3 FOLLOWER]: Leader election won for term 3
I20250110 20:53:32.877905 5222 raft_consensus.cc:695] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [term 3 LEADER]: Becoming Leader. State: Replica: 5137413ae3884ca68ee8f187db732e8e, State: Running, Role: LEADER
I20250110 20:53:32.878335 5222 consensus_queue.cc:237] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:32.881739 4399 catalog_manager.cc:5526] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e reported cstate change: term changed from 2 to 3, leader changed from 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4) to 5137413ae3884ca68ee8f187db732e8e (127.0.104.3). New cstate: current_term: 3 leader_uuid: "5137413ae3884ca68ee8f187db732e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } health_report { overall_health: UNKNOWN } } }
I20250110 20:53:33.000316 416 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20250110 20:53:33.003731 5112 meta_cache.cc:302] tablet 58912c24a6f040dcb3ab3ff65a0c1cd3: replica 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075) has failed: Network error: Client connection negotiation failed: client connection to 127.0.104.4:36075: connect: Connection refused (error 111)
W20250110 20:53:33.013401 4633 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60598: Illegal state: replica 9902c01dcf24446aa6c1f6c17aba77d0 is not leader of this config: current role FOLLOWER
I20250110 20:53:33.026914 5063 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5d21cd10ee8f4606aee63432a45ec7c3 [term 3 FOLLOWER]: Refusing update from remote peer 5137413ae3884ca68ee8f187db732e8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250110 20:53:33.027292 4672 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9902c01dcf24446aa6c1f6c17aba77d0 [term 3 FOLLOWER]: Refusing update from remote peer 5137413ae3884ca68ee8f187db732e8e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250110 20:53:33.027611 5233 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250110 20:53:33.027949 4737 consensus_peers.cc:487] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e -> Peer 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075): Couldn't send request to peer 9860e3170fbc43c0b7d4fc77e60c3490. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.4:36075: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:53:33.028112 5222 consensus_queue.cc:1035] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250110 20:53:33.028342 4739 consensus_peers.cc:487] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e -> Peer fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597): Couldn't send request to peer fed858ce3e3d428ea64e46e861c8d5b7. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:36597: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:53:33.067708 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.4:36075
--local_ip_for_outbound_sockets=127.0.104.4
--tserver_master_addrs=127.0.104.62:42769
--webserver_port=35211
--webserver_interface=127.0.104.4
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20250110 20:53:33.388733 5254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:33.389031 5254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:33.389145 5254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:33.397002 5254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:33.397207 5254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.4
I20250110 20:53:33.406571 5254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.4:36075
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.104.4
--webserver_port=35211
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.4
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:33.408185 5254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:33.409929 5254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:33.421252 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:33.421214 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:33.421274 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:33.422137 5254 server_base.cc:1034] running on GCE node
I20250110 20:53:33.422727 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:33.423709 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:33.424959 5254 hybrid_clock.cc:648] HybridClock initialized: now 1736542413424875 us; error 42 us; skew 500 ppm
I20250110 20:53:33.425261 5254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:33.431355 5254 webserver.cc:458] Webserver started at http://127.0.104.4:35211/ using document root <none> and password file <none>
I20250110 20:53:33.432152 5254 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:33.432291 5254 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:33.437692 5254 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20250110 20:53:33.440564 5268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:33.441890 5254 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:53:33.442131 5254 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "9860e3170fbc43c0b7d4fc77e60c3490"
format_stamp: "Formatted at 2025-01-10 20:53:28 on dist-test-slave-1h8z"
I20250110 20:53:33.443032 5254 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:33.466828 5254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:33.467669 5254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:33.467914 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:33.468811 5254 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:33.470649 5275 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250110 20:53:33.475795 5254 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250110 20:53:33.475929 5254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20250110 20:53:33.476039 5254 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250110 20:53:33.479036 5254 ts_tablet_manager.cc:610] Registered 1 tablets
I20250110 20:53:33.479135 5254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:33.479489 5275 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Bootstrap starting.
I20250110 20:53:33.502918 5275 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:33.516328 5254 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.4:36075
I20250110 20:53:33.516348 5382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.4:36075 every 8 connection(s)
I20250110 20:53:33.517736 5254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 20:53:33.527793 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 5254
I20250110 20:53:33.528096 5275 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 20:53:33.528417 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:36597
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:42769
--webserver_port=40979
--webserver_interface=127.0.104.1
--builtin_ntp_servers=127.0.104.20:45571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250110 20:53:33.528739 5275 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Bootstrap complete.
I20250110 20:53:33.530084 5275 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Time spent bootstrapping tablet: real 0.051s user 0.033s sys 0.015s
I20250110 20:53:33.532639 5383 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:33.533020 5383 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:33.534094 5383 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:33.536295 5275 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:33.537174 5275 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9860e3170fbc43c0b7d4fc77e60c3490, State: Initialized, Role: FOLLOWER
I20250110 20:53:33.537331 4399 ts_manager.cc:194] Re-registered known tserver with Master: 9860e3170fbc43c0b7d4fc77e60c3490 (127.0.104.4:36075)
I20250110 20:53:33.537850 5275 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:33.538806 4399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.4:40405
I20250110 20:53:33.539454 5275 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250110 20:53:33.541275 5383 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
I20250110 20:53:33.578228 5337 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 9860e3170fbc43c0b7d4fc77e60c3490 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:33.587427 5390 mvcc.cc:204] Tried to move back new op lower bound from 7112877723750649856 to 7112877723154149376. Current Snapshot: MvccSnapshot[applied={T|T < 7112877723750649856}]
W20250110 20:53:33.858795 5387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:33.859107 5387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:33.859234 5387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:33.867095 5387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:33.867295 5387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:33.876646 5387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:36597
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=40979
--tserver_master_addrs=127.0.104.62:42769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:33.878216 5387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:33.879932 5387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:33.891180 5400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:33.891238 5399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:33.891217 5402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:33.892086 5387 server_base.cc:1034] running on GCE node
I20250110 20:53:33.892643 5387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:33.893543 5387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:33.894817 5387 hybrid_clock.cc:648] HybridClock initialized: now 1736542413894696 us; error 50 us; skew 500 ppm
I20250110 20:53:33.895118 5387 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:33.897387 5387 webserver.cc:458] Webserver started at http://127.0.104.1:40979/ using document root <none> and password file <none>
I20250110 20:53:33.898133 5387 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:33.898259 5387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:33.903535 5387 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250110 20:53:33.906324 5408 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:33.907740 5387 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:33.907905 5387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fed858ce3e3d428ea64e46e861c8d5b7"
format_stamp: "Formatted at 2025-01-10 20:53:27 on dist-test-slave-1h8z"
I20250110 20:53:33.908713 5387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:33.925336 5387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:33.926194 5387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:33.926445 5387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:33.927413 5387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:33.929301 5415 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250110 20:53:33.934433 5387 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250110 20:53:33.934564 5387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20250110 20:53:33.934679 5387 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250110 20:53:33.937573 5387 ts_tablet_manager.cc:610] Registered 1 tablets
I20250110 20:53:33.937664 5387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20250110 20:53:33.938028 5415 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Bootstrap starting.
I20250110 20:53:33.958250 5415 log.cc:826] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:33.977741 5415 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 20:53:33.978440 5415 tablet_bootstrap.cc:492] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Bootstrap complete.
I20250110 20:53:33.979710 5415 ts_tablet_manager.cc:1397] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Time spent bootstrapping tablet: real 0.042s user 0.024s sys 0.016s
I20250110 20:53:33.981103 5387 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36597
I20250110 20:53:33.981137 5522 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36597 every 8 connection(s)
I20250110 20:53:33.982462 5387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:33.986615 5415 raft_consensus.cc:357] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:33.987576 5415 raft_consensus.cc:738] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fed858ce3e3d428ea64e46e861c8d5b7, State: Initialized, Role: FOLLOWER
I20250110 20:53:33.988234 5415 consensus_queue.cc:260] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9860e3170fbc43c0b7d4fc77e60c3490" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 36075 } } peers { permanent_uuid: "9902c01dcf24446aa6c1f6c17aba77d0" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36031 } } peers { permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 } } peers { permanent_uuid: "5137413ae3884ca68ee8f187db732e8e" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44051 } } peers { permanent_uuid: "5d21cd10ee8f4606aee63432a45ec7c3" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 42353 } }
I20250110 20:53:33.989645 5415 ts_tablet_manager.cc:1428] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20250110 20:53:33.990818 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 5387
I20250110 20:53:33.994668 5523 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42769
I20250110 20:53:33.995014 5523 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:33.995913 5523 heartbeater.cc:510] Master 127.0.104.62:42769 requested a full tablet report, sending...
I20250110 20:53:33.998428 4399 ts_manager.cc:194] Re-registered known tserver with Master: fed858ce3e3d428ea64e46e861c8d5b7 (127.0.104.1:36597)
I20250110 20:53:33.999889 4399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:53919
I20250110 20:53:34.002646 5523 heartbeater.cc:502] Master 127.0.104.62:42769 was elected leader, sending a full tablet report...
I20250110 20:53:34.068641 5478 raft_consensus.cc:3054] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 1 FOLLOWER]: Advancing to term 3
I20250110 20:53:34.080930 5478 raft_consensus.cc:1270] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 3 FOLLOWER]: Refusing update from remote peer 5137413ae3884ca68ee8f187db732e8e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250110 20:53:34.092041 5317 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:53:34.099385 5556 consensus_queue.cc:1037] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P 5137413ae3884ca68ee8f187db732e8e [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "fed858ce3e3d428ea64e46e861c8d5b7" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20250110 20:53:34.113169 5043 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:53:34.127202 4653 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250110 20:53:34.143569 4783 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250110 20:53:34.168794 5477 raft_consensus.cc:1212] T 58912c24a6f040dcb3ab3ff65a0c1cd3 P fed858ce3e3d428ea64e46e861c8d5b7 [term 3 FOLLOWER]: Deduplicated request from leader. Original: 1.6->[2.7-3.18] Dedup: 3.18->[]
W20250110 20:53:34.169571 5478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:47761 (request call id 1) took 68 ms (client timeout 50 ms). Trace:
I20250110 20:53:34.170666 5457 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20250110 20:53:34.169687 5478 rpcz_store.cc:269] 0110 20:53:34.100987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:34.101098 (+ 111us) service_pool.cc:225] Handling call
0110 20:53:34.101499 (+ 401us) raft_consensus.cc:1466] Updating replica for 12 ops
0110 20:53:34.101943 (+ 444us) raft_consensus.cc:1529] Early marking committed up to index 0
0110 20:53:34.101953 (+ 10us) raft_consensus.cc:1534] Triggering prepare for 12 ops
0110 20:53:34.102938 (+ 985us) write_op.cc:270] Start()
0110 20:53:34.103006 (+ 68us) write_op.cc:276] Timestamp: P: 1736542411297574 usec, L: 0
0110 20:53:34.106064 (+ 3058us) write_op.cc:270] Start()
0110 20:53:34.106102 (+ 38us) write_op.cc:276] Timestamp: P: 1736542411320922 usec, L: 0
0110 20:53:34.106347 (+ 245us) write_op.cc:270] Start()
0110 20:53:34.106384 (+ 37us) write_op.cc:276] Timestamp: P: 1736542411329170 usec, L: 0
0110 20:53:34.106555 (+ 171us) write_op.cc:270] Start()
0110 20:53:34.106576 (+ 21us) write_op.cc:276] Timestamp: P: 1736542411342161 usec, L: 0
0110 20:53:34.106775 (+ 199us) write_op.cc:270] Start()
0110 20:53:34.106798 (+ 23us) write_op.cc:276] Timestamp: P: 1736542411348504 usec, L: 0
0110 20:53:34.106964 (+ 166us) write_op.cc:270] Start()
0110 20:53:34.106983 (+ 19us) write_op.cc:276] Timestamp: P: 1736542413025061 usec, L: 0
0110 20:53:34.107106 (+ 123us) write_op.cc:270] Start()
0110 20:53:34.107186 (+ 80us) write_op.cc:276] Timestamp: P: 1736542413038039 usec, L: 0
0110 20:53:34.107329 (+ 143us) write_op.cc:270] Start()
0110 20:53:34.107348 (+ 19us) write_op.cc:276] Timestamp: P: 1736542413045656 usec, L: 0
0110 20:53:34.107523 (+ 175us) write_op.cc:270] Start()
0110 20:53:34.107543 (+ 20us) write_op.cc:276] Timestamp: P: 1736542413057094 usec, L: 0
0110 20:53:34.107690 (+ 147us) write_op.cc:270] Start()
0110 20:53:34.107710 (+ 20us) write_op.cc:276] Timestamp: P: 1736542413063115 usec, L: 0
0110 20:53:34.108488 (+ 778us) log.cc:844] Serialized 6732 byte log entry
0110 20:53:34.118078 (+ 9590us) raft_consensus.cc:1640] Marking committed up to 18
0110 20:53:34.118304 (+ 226us) log.cc:844] Serialized 14 byte log entry
0110 20:53:34.121974 (+ 3670us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.121979 (+ 5us) write_op.cc:454] Released schema lock
0110 20:53:34.163697 (+ 41718us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.163706 (+ 9us) write_op.cc:454] Released schema lock
0110 20:53:34.164012 (+ 306us) log.cc:844] Serialized 14 byte log entry
0110 20:53:34.166235 (+ 2223us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.166239 (+ 4us) write_op.cc:454] Released schema lock
0110 20:53:34.166293 (+ 54us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:53:34.168600 (+ 2307us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:53:34.168632 (+ 32us) raft_consensus.cc:1680] finished
0110 20:53:34.168637 (+ 5us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:53:34.169505 (+ 868us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:34.104569 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.104771 (+ 202us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.104779 (+ 8us) write_op.cc:435] Acquired schema lock
0110 20:53:34.104781 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:34.105102 (+ 321us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.105123 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.105128 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.105864 (+ 736us) tablet.cc:698] Row locks acquired
0110 20:53:34.105867 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:34.118486 (+ 12619us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.119444 (+ 958us) write_op.cc:301] APPLY: starting
0110 20:53:34.119467 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.119784 (+ 317us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.119787 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.120124 (+ 337us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.120338 (+ 214us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.120361 (+ 23us) write_op.cc:312] APPLY: finished
0110 20:53:34.120535 (+ 174us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.120665 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.120694 (+ 29us) write_op.cc:454] Released schema lock
0110 20:53:34.120706 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.120966 (+ 260us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.120971 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.107828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.107936 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.107942 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:53:34.107945 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:34.108100 (+ 155us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.108128 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.108133 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.108294 (+ 161us) tablet.cc:698] Row locks acquired
0110 20:53:34.108297 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:34.118907 (+ 10610us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.121071 (+ 2164us) write_op.cc:301] APPLY: starting
0110 20:53:34.121084 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.121132 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.121135 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.121413 (+ 278us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.121464 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.121471 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:34.121548 (+ 77us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.121606 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.121625 (+ 19us) write_op.cc:454] Released schema lock
0110 20:53:34.121631 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 20:53:34.108408 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.108490 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.108495 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:34.108498 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:34.108922 (+ 424us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.108937 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.108941 (+ 4us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.109059 (+ 118us) tablet.cc:698] Row locks acquired
0110 20:53:34.109062 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:34.122065 (+ 13003us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.124137 (+ 2072us) write_op.cc:301] APPLY: starting
0110 20:53:34.124153 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.124206 (+ 53us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.124209 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.124407 (+ 198us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.124465 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.124472 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:34.124564 (+ 92us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.124629 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.124651 (+ 22us) write_op.cc:454] Released schema lock
0110 20:53:34.124658 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.124756 (+ 98us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.124760 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.109156 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.109243 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.109249 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:53:34.109251 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:34.109397 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.109409 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.109414 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.109503 (+ 89us) tablet.cc:698] Row locks acquired
0110 20:53:34.109506 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:34.122182 (+ 12676us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.124849 (+ 2667us) write_op.cc:301] APPLY: starting
0110 20:53:34.124861 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.124911 (+ 50us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.124913 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.125106 (+ 193us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.125159 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.125166 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:34.125262 (+ 96us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.125307 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.125325 (+ 18us) write_op.cc:454] Released schema lock
0110 20:53:34.125331 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.125416 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.125420 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.109575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.109641 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.109645 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:53:34.109648 (+ 3us) tablet.cc:659] Decoding operations
0110 20:53:34.109763 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.109772 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.109777 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.109859 (+ 82us) tablet.cc:698] Row locks acquired
0110 20:53:34.109861 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:34.122218 (+ 12357us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.125490 (+ 3272us) write_op.cc:301] APPLY: starting
0110 20:53:34.125500 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.125543 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.125546 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.125740 (+ 194us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.125793 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.125801 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:34.125876 (+ 75us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.125930 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.125947 (+ 17us) write_op.cc:454] Released schema lock
0110 20:53:34.125952 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0110 20:53:34.109929 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.109999 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.110004 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:34.110006 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:34.110140 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.110150 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.110155 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.110233 (+ 78us) tablet.cc:698] Row locks acquired
0110 20:53:34.110235 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:34.164203 (+ 53968us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.164305 (+ 102us) write_op.cc:301] APPLY: starting
0110 20:53:34.164323 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.164450 (+ 127us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.164453 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.164988 (+ 535us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.165086 (+ 98us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.165094 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:34.165190 (+ 96us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.165262 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.165277 (+ 15us) write_op.cc:454] Released schema lock
0110 20:53:34.165283 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.165346 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.165349 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.110307 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.110377 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.110382 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:34.110387 (+ 5us) tablet.cc:659] Decoding operations
0110 20:53:34.110518 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.110528 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.110533 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.110629 (+ 96us) tablet.cc:698] Row locks acquired
0110 20:53:34.110632 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:34.164280 (+ 53648us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.164386 (+ 106us) write_op.cc:301] APPLY: starting
0110 20:53:34.164395 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.164449 (+ 54us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.164452 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.164798 (+ 346us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.164891 (+ 93us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.164902 (+ 11us) write_op.cc:312] APPLY: finished
0110 20:53:34.165003 (+ 101us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.165076 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.165100 (+ 24us) write_op.cc:454] Released schema lock
0110 20:53:34.165108 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.165229 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.165234 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.110699 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.110787 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.110792 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:34.110794 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:34.110901 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.110914 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.110918 (+ 4us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.110986 (+ 68us) tablet.cc:698] Row locks acquired
0110 20:53:34.110988 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:34.164326 (+ 53338us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.164446 (+ 120us) write_op.cc:301] APPLY: starting
0110 20:53:34.164465 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.164585 (+ 120us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.164587 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.164944 (+ 357us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.165312 (+ 368us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.165321 (+ 9us) write_op.cc:312] APPLY: finished
0110 20:53:34.165453 (+ 132us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.165611 (+ 158us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.165631 (+ 20us) write_op.cc:454] Released schema lock
0110 20:53:34.165638 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.165714 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.165718 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.111079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.111156 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.111160 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:53:34.111162 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:34.111295 (+ 133us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.111307 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.111311 (+ 4us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.111425 (+ 114us) tablet.cc:698] Row locks acquired
0110 20:53:34.111428 (+ 3us) write_op.cc:260] PREPARE: finished
0110 20:53:34.165383 (+ 53955us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.165475 (+ 92us) write_op.cc:301] APPLY: starting
0110 20:53:34.165486 (+ 11us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.165533 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.165536 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.165907 (+ 371us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.165960 (+ 53us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.165966 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:53:34.166033 (+ 67us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.166098 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.166114 (+ 16us) write_op.cc:454] Released schema lock
0110 20:53:34.166120 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0110 20:53:34.166190 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.166194 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0110 20:53:34.111539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
0110 20:53:34.111627 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:34.111632 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:53:34.111634 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:34.111758 (+ 124us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:34.111769 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:34.111774 (+ 5us) tablet.cc:682] Acquiring locks for 10 operations
0110 20:53:34.111855 (+ 81us) tablet.cc:698] Row locks acquired
0110 20:53:34.111857 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:34.165465 (+ 53608us) op_driver.cc:464] REPLICATION: finished
0110 20:53:34.165522 (+ 57us) write_op.cc:301] APPLY: starting
0110 20:53:34.165530 (+ 8us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:34.165571 (+ 41us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:34.165574 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:34.165915 (+ 341us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:34.165957 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:34.165964 (+ 7us) write_op.cc:312] APPLY: finished
0110 20:53:34.166033 (+ 69us) log.cc:844] Serialized 76 byte log entry
0110 20:53:34.166117 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:34.166132 (+ 15us) write_op.cc:454] Released schema lock
0110 20:53:34.166137 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"prepare.queue_time_us":118,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":33,"raft.queue_time_us":129,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":105,"thread_start_us":9033,"threads_started":1,"wal-append.queue_time_us":7902,"child_traces":[["op",{"apply.queue_time_us":434,"apply.run_cpu_time_us":1508,"apply.run_wall_time_us":1507,"num_ops":10,"partition_lock_wait_us":10,"prepare.queue_time_us":363,"prepare.run_cpu_time_us":863,"prepare.run_wall_time_us":1455,"replication_time_us":15517,"thread_start_us":1630,"threads_started":2}],["op",{"apply.queue_time_us":2119,"apply.run_cpu_time_us":634,"apply.run_wall_time_us":634,"mutex_wait_us":43,"num_ops":10,"partition_lock_wait_us":18,"prepare.queue_time_us":1602,"prepare.run_cpu_time_us":541,"prepare.run_wall_time_us":541,"replication_time_us":12824,"thread_start_us":2951,"threads_started":1}],["op",{"apply.queue_time_us":2010,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":611,"mutex_wait_us":47,"num_ops":10,"partition_lock_wait_us":5,"prepare.queue_time_us":1940,"prepare.run_cpu_time_us":437,"prepare.run_wall_time_us":707,"replication_time_us":15703}],["op",{"apply.queue_time_us":2634,"apply.run_cpu_time_us":557,"apply.run_wall_time_us":556,"num_ops":10,"partition_lock_wait_us":4,"prepare.queue_time_us":2517,"prepare.run_cpu_time_us":390,"prepare.run_wall_time_us":390,"replication_time_us":15616}],["op",{"apply.queue_time_us":3243,"apply.run_cpu_time_us":520,"apply.run_wall_time_us":520,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":2707,"prepare.run_cpu_time_us":324,"prepare.run_wall_time_us":324,"replication_time_us":15436,"thread_start_us":41407,"threads_started":1}],["op",{"apply.queue_time_us":26,"apply.run_cpu_time_us":933,"apply.run_wall_time_us":1037,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":2893,"prepare.run_cpu_time_us":343,"prepare.run_wall_time_us":342,"replication_time_us":57211,"spinlock_wait_cycles":384}],["op",{"apply.queue_time_us":75,"apply.run_cpu_time_us":826,"apply.run_wall_time_us":825,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":3058,"prepare.run_cpu_time_us":363,"prepare.run_wall_time_us":363,"replication_time_us":57158,"spinlock_wait_cycles":1152}],["op",{"apply.queue_time_us":38,"apply.run_cpu_time_us":897,"apply.run_wall_time_us":1268,"num_ops":10,"partition_lock_wait_us":2,"prepare.queue_time_us":3284,"prepare.run_cpu_time_us":330,"prepare.run_wall_time_us":329,"replication_time_us":56987}],["op",{"apply.queue_time_us":48,"apply.run_cpu_time_us":707,"apply.run_wall_time_us":706,"num_ops":10,"partition_lock_wait_us":4,"prepare.queue_time_us":3463,"prepare.run_cpu_time_us":401,"prepare.run_wall_time_us":400,"replication_time_us":57840}],["op",{"apply.queue_time_us":28,"apply.run_cpu_time_us":674,"apply.run_wall_time_us":674,"mutex_wait_us":3,"num_ops":10,"partition_lock_wait_us":3,"prepare.queue_time_us":3765,"prepare.run_cpu_time_us":366,"prepare.run_wall_time_us":365,"replication_time_us":57762,"spinlock_wait_cycles":34560}]]}
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
85367fd4e73f456e9629664973cc4fb8 | 127.0.104.62:42769 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:45571 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
5137413ae3884ca68ee8f187db732e8e | 127.0.104.3:44051 | HEALTHY | <none> | 1 | 0
5d21cd10ee8f4606aee63432a45ec7c3 | 127.0.104.5:42353 | HEALTHY | <none> | 0 | 0
9860e3170fbc43c0b7d4fc77e60c3490 | 127.0.104.4:36075 | HEALTHY | <none> | 0 | 0
9902c01dcf24446aa6c1f6c17aba77d0 | 127.0.104.2:36031 | HEALTHY | <none> | 0 | 0
fed858ce3e3d428ea64e46e861c8d5b7 | 127.0.104.1:36597 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.0.104.1 | experimental | 127.0.104.1:36597
local_ip_for_outbound_sockets | 127.0.104.2 | experimental | 127.0.104.2:36031
local_ip_for_outbound_sockets | 127.0.104.3 | experimental | 127.0.104.3:44051
local_ip_for_outbound_sockets | 127.0.104.4 | experimental | 127.0.104.4:36075
local_ip_for_outbound_sockets | 127.0.104.5 | experimental | 127.0.104.5:42353
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.0.104.1:36597
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.0.104.2:36031
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.0.104.3:44051
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.0.104.4:36075
server_dump_info_path | /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.0.104.5:42353
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.0.104.20:45571 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250110 20:53:34.210275 416 log_verifier.cc:126] Checking tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
I20250110 20:53:34.259963 416 log_verifier.cc:177] Verified matching terms for 18 ops in tablet 58912c24a6f040dcb3ab3ff65a0c1cd3
I20250110 20:53:34.523991 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 5387
I20250110 20:53:34.531416 5518 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:34.669700 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 5387
I20250110 20:53:34.683686 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 4591
I20250110 20:53:34.684880 4713 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:34.801185 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 4591
I20250110 20:53:34.817171 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 4721
I20250110 20:53:34.818137 4843 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:34.910631 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 4721
I20250110 20:53:34.930893 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 5254
I20250110 20:53:34.937865 5377 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:35.034286 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 5254
I20250110 20:53:35.048483 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 4981
I20250110 20:53:35.049415 5103 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:35.139180 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 4981
I20250110 20:53:35.152846 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 4370
I20250110 20:53:35.153848 4431 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:35.259691 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 4370
2025-01-10T20:53:35Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (9040 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20250110 20:53:35.301455 416 test_util.cc:274] Using random seed: 1238182696
I20250110 20:53:35.303185 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:53:35.303272 416 ts_itest-base.cc:113] --------------
I20250110 20:53:35.303320 416 ts_itest-base.cc:114] 1 tablet servers
I20250110 20:53:35.303378 416 ts_itest-base.cc:115] 1 replicas per TS
I20250110 20:53:35.303424 416 ts_itest-base.cc:116] --------------
2025-01-10T20:53:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:53:35Z Disabled control of system clock
I20250110 20:53:35.332031 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41313
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41313 with env {}
W20250110 20:53:35.653942 5587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:35.654280 5587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:35.654404 5587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:35.662492 5587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:35.662623 5587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:35.662694 5587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:35.662811 5587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:35.672348 5587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41313
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:35.673947 5587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:35.675724 5587 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:35.686030 5593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:35.686004 5592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:35.686074 5595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:35.686970 5587 server_base.cc:1034] running on GCE node
I20250110 20:53:35.687723 5587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:35.689157 5587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:35.690366 5587 hybrid_clock.cc:648] HybridClock initialized: now 1736542415690299 us; error 39 us; skew 500 ppm
I20250110 20:53:35.690701 5587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:35.692996 5587 webserver.cc:458] Webserver started at http://127.0.104.62:33439/ using document root <none> and password file <none>
I20250110 20:53:35.693711 5587 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:35.693821 5587 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:35.694137 5587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:35.696774 5587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8e1a175249534df191467ce1f3879d51"
format_stamp: "Formatted at 2025-01-10 20:53:35 on dist-test-slave-1h8z"
I20250110 20:53:35.697522 5587 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8e1a175249534df191467ce1f3879d51"
format_stamp: "Formatted at 2025-01-10 20:53:35 on dist-test-slave-1h8z"
I20250110 20:53:35.702644 5587 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250110 20:53:35.705978 5601 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:35.707329 5587 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:35.707490 5587 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8e1a175249534df191467ce1f3879d51"
format_stamp: "Formatted at 2025-01-10 20:53:35 on dist-test-slave-1h8z"
I20250110 20:53:35.707669 5587 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:35.726155 5587 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:35.727033 5587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:35.727319 5587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:35.743494 5587 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41313
I20250110 20:53:35.743512 5652 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41313 every 8 connection(s)
I20250110 20:53:35.744835 5587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:35.748466 5653 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:35.751596 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 5587
I20250110 20:53:35.751817 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:53:35.758891 5653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51: Bootstrap starting.
I20250110 20:53:35.761969 5653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:35.763077 5653 log.cc:826] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:35.765826 5653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51: No bootstrap required, opened a new log
I20250110 20:53:35.772135 5653 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1a175249534df191467ce1f3879d51" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41313 } }
I20250110 20:53:35.772503 5653 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:35.772583 5653 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e1a175249534df191467ce1f3879d51, State: Initialized, Role: FOLLOWER
I20250110 20:53:35.773180 5653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1a175249534df191467ce1f3879d51" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41313 } }
I20250110 20:53:35.773367 5653 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:35.773460 5653 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:35.773578 5653 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:35.775591 5653 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1a175249534df191467ce1f3879d51" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41313 } }
I20250110 20:53:35.776053 5653 leader_election.cc:304] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8e1a175249534df191467ce1f3879d51; no voters:
I20250110 20:53:35.776607 5653 leader_election.cc:290] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:35.776810 5658 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:35.777783 5658 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [term 1 LEADER]: Becoming Leader. State: Replica: 8e1a175249534df191467ce1f3879d51, State: Running, Role: LEADER
I20250110 20:53:35.778260 5658 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1a175249534df191467ce1f3879d51" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41313 } }
I20250110 20:53:35.778651 5653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:35.780953 5660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8e1a175249534df191467ce1f3879d51. Latest consensus state: current_term: 1 leader_uuid: "8e1a175249534df191467ce1f3879d51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1a175249534df191467ce1f3879d51" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41313 } } }
I20250110 20:53:35.781252 5660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:35.781774 5659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8e1a175249534df191467ce1f3879d51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8e1a175249534df191467ce1f3879d51" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41313 } } }
I20250110 20:53:35.782034 5659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:35.791390 5665 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:35.796864 5665 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:35.805557 5665 catalog_manager.cc:1348] Generated new cluster ID: 4044bfcdf281454e9793e3aacb6a7897
I20250110 20:53:35.805677 5665 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:35.816059 5665 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:53:35.817281 5665 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:35.829963 5665 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8e1a175249534df191467ce1f3879d51: Generated new TSK 0
I20250110 20:53:35.831096 5665 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:53:35.843012 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41313
--builtin_ntp_servers=127.0.104.20:36317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20250110 20:53:36.184181 5678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:36.184476 5678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:36.184592 5678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:36.192345 5678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:36.192574 5678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:36.202229 5678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41313
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:36.203892 5678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:36.205602 5678 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:36.216823 5683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:36.216922 5686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:36.216922 5684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:36.217993 5678 server_base.cc:1034] running on GCE node
I20250110 20:53:36.218600 5678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:36.219571 5678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:36.220800 5678 hybrid_clock.cc:648] HybridClock initialized: now 1736542416220727 us; error 44 us; skew 500 ppm
I20250110 20:53:36.221107 5678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:36.223346 5678 webserver.cc:458] Webserver started at http://127.0.104.1:45321/ using document root <none> and password file <none>
I20250110 20:53:36.224092 5678 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:36.224208 5678 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:36.224507 5678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:36.227154 5678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9461e8449a4f47dcaa850e8c73dd8f94"
format_stamp: "Formatted at 2025-01-10 20:53:36 on dist-test-slave-1h8z"
I20250110 20:53:36.227864 5678 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9461e8449a4f47dcaa850e8c73dd8f94"
format_stamp: "Formatted at 2025-01-10 20:53:36 on dist-test-slave-1h8z"
I20250110 20:53:36.232969 5678 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:53:36.236366 5692 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:36.237772 5678 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250110 20:53:36.237942 5678 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9461e8449a4f47dcaa850e8c73dd8f94"
format_stamp: "Formatted at 2025-01-10 20:53:36 on dist-test-slave-1h8z"
I20250110 20:53:36.238137 5678 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:36.268432 5678 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:36.269261 5678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:36.269520 5678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:36.270433 5678 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:36.271936 5678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:36.272013 5678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:36.272141 5678 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:36.272192 5678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:36.301152 5678 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36171
I20250110 20:53:36.301198 5804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36171 every 8 connection(s)
I20250110 20:53:36.302433 5678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:36.305799 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 5678
I20250110 20:53:36.305968 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:53:36.315387 5805 heartbeater.cc:346] Connected to a master server at 127.0.104.62:41313
I20250110 20:53:36.315701 5805 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:36.316499 5805 heartbeater.cc:510] Master 127.0.104.62:41313 requested a full tablet report, sending...
I20250110 20:53:36.318485 5618 ts_manager.cc:194] Registered new tserver with Master: 9461e8449a4f47dcaa850e8c73dd8f94 (127.0.104.1:36171)
I20250110 20:53:36.320158 5618 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:47137
I20250110 20:53:36.321959 416 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250110 20:53:36.336472 5618 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41354:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250110 20:53:36.362582 5740 tablet_service.cc:1467] Processing CreateTablet for tablet 184a1b12e84742d884a437daa49aaa57 (DEFAULT_TABLE table=TestTable [id=a58f4dd145b6467b997fe29a77231788]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:36.363909 5740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 184a1b12e84742d884a437daa49aaa57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:36.373718 5819 tablet_bootstrap.cc:492] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94: Bootstrap starting.
I20250110 20:53:36.376981 5819 tablet_bootstrap.cc:654] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:36.377974 5819 log.cc:826] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:36.380424 5819 tablet_bootstrap.cc:492] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94: No bootstrap required, opened a new log
I20250110 20:53:36.380673 5819 ts_tablet_manager.cc:1397] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:53:36.386929 5819 raft_consensus.cc:357] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9461e8449a4f47dcaa850e8c73dd8f94" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36171 } }
I20250110 20:53:36.387308 5819 raft_consensus.cc:383] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:36.387401 5819 raft_consensus.cc:738] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9461e8449a4f47dcaa850e8c73dd8f94, State: Initialized, Role: FOLLOWER
I20250110 20:53:36.387974 5819 consensus_queue.cc:260] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9461e8449a4f47dcaa850e8c73dd8f94" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36171 } }
I20250110 20:53:36.388180 5819 raft_consensus.cc:397] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:36.388294 5819 raft_consensus.cc:491] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:36.388432 5819 raft_consensus.cc:3054] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:36.390389 5819 raft_consensus.cc:513] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9461e8449a4f47dcaa850e8c73dd8f94" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36171 } }
I20250110 20:53:36.390906 5819 leader_election.cc:304] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9461e8449a4f47dcaa850e8c73dd8f94; no voters:
I20250110 20:53:36.391541 5819 leader_election.cc:290] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:36.391685 5821 raft_consensus.cc:2798] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:36.392756 5805 heartbeater.cc:502] Master 127.0.104.62:41313 was elected leader, sending a full tablet report...
I20250110 20:53:36.392952 5821 raft_consensus.cc:695] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [term 1 LEADER]: Becoming Leader. State: Replica: 9461e8449a4f47dcaa850e8c73dd8f94, State: Running, Role: LEADER
I20250110 20:53:36.393497 5819 ts_tablet_manager.cc:1428] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20250110 20:53:36.393473 5821 consensus_queue.cc:237] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9461e8449a4f47dcaa850e8c73dd8f94" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36171 } }
I20250110 20:53:36.398520 5618 catalog_manager.cc:5526] T 184a1b12e84742d884a437daa49aaa57 P 9461e8449a4f47dcaa850e8c73dd8f94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9461e8449a4f47dcaa850e8c73dd8f94 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "9461e8449a4f47dcaa850e8c73dd8f94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9461e8449a4f47dcaa850e8c73dd8f94" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36171 } health_report { overall_health: HEALTHY } } }
I20250110 20:53:36.407789 416 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250110 20:53:36.409595 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9461e8449a4f47dcaa850e8c73dd8f94 to finish bootstrapping
I20250110 20:53:36.424839 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 5678
I20250110 20:53:36.425896 5800 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:36.510313 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 5678
I20250110 20:53:36.523905 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 5587
I20250110 20:53:36.524995 5648 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:36.624498 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 5587
2025-01-10T20:53:36Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (1358 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20250110 20:53:36.660241 416 test_util.cc:274] Using random seed: 1239541495
I20250110 20:53:36.661938 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:53:36.662003 416 ts_itest-base.cc:113] --------------
I20250110 20:53:36.662056 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:53:36.662103 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:53:36.662163 416 ts_itest-base.cc:116] --------------
2025-01-10T20:53:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:53:36Z Disabled control of system clock
I20250110 20:53:36.691382 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33169
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:33169
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250110 20:53:37.013731 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:37.014055 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:37.014181 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:37.022424 5837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:37.022552 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:37.022611 5837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:37.022683 5837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:37.032552 5837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:33169
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:33169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:37.034164 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:37.035885 5837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:37.046006 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:37.046002 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:37.046058 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:37.046590 5837 server_base.cc:1034] running on GCE node
I20250110 20:53:37.047577 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:37.048965 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:37.050199 5837 hybrid_clock.cc:648] HybridClock initialized: now 1736542417050138 us; error 36 us; skew 500 ppm
I20250110 20:53:37.050529 5837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:37.052793 5837 webserver.cc:458] Webserver started at http://127.0.104.62:42535/ using document root <none> and password file <none>
I20250110 20:53:37.053503 5837 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:37.053619 5837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:37.053925 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:37.056540 5837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a26a7b6e65b7411d9f3b2721ce7b0027"
format_stamp: "Formatted at 2025-01-10 20:53:37 on dist-test-slave-1h8z"
I20250110 20:53:37.057300 5837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a26a7b6e65b7411d9f3b2721ce7b0027"
format_stamp: "Formatted at 2025-01-10 20:53:37 on dist-test-slave-1h8z"
I20250110 20:53:37.062379 5837 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250110 20:53:37.065726 5851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:37.067124 5837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:37.067287 5837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a26a7b6e65b7411d9f3b2721ce7b0027"
format_stamp: "Formatted at 2025-01-10 20:53:37 on dist-test-slave-1h8z"
I20250110 20:53:37.067461 5837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:37.089814 5837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:37.090670 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:37.090988 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:37.107463 5837 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:33169
I20250110 20:53:37.107486 5902 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:33169 every 8 connection(s)
I20250110 20:53:37.108793 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:37.111536 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 5837
I20250110 20:53:37.111703 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:53:37.112567 5903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:37.124677 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027: Bootstrap starting.
I20250110 20:53:37.127823 5903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:37.128803 5903 log.cc:826] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:37.131609 5903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027: No bootstrap required, opened a new log
I20250110 20:53:37.137956 5903 raft_consensus.cc:357] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33169 } }
I20250110 20:53:37.138314 5903 raft_consensus.cc:383] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:37.138428 5903 raft_consensus.cc:738] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a26a7b6e65b7411d9f3b2721ce7b0027, State: Initialized, Role: FOLLOWER
I20250110 20:53:37.139083 5903 consensus_queue.cc:260] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33169 } }
I20250110 20:53:37.139292 5903 raft_consensus.cc:397] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:37.139430 5903 raft_consensus.cc:491] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:37.139565 5903 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:37.141667 5903 raft_consensus.cc:513] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33169 } }
I20250110 20:53:37.142215 5903 leader_election.cc:304] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a26a7b6e65b7411d9f3b2721ce7b0027; no voters:
I20250110 20:53:37.142874 5903 leader_election.cc:290] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:37.143005 5908 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:37.143589 5908 raft_consensus.cc:695] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [term 1 LEADER]: Becoming Leader. State: Replica: a26a7b6e65b7411d9f3b2721ce7b0027, State: Running, Role: LEADER
I20250110 20:53:37.144097 5908 consensus_queue.cc:237] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33169 } }
I20250110 20:53:37.144860 5903 sys_catalog.cc:564] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:37.146925 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33169 } } }
I20250110 20:53:37.146958 5910 sys_catalog.cc:455] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a26a7b6e65b7411d9f3b2721ce7b0027. Latest consensus state: current_term: 1 leader_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a26a7b6e65b7411d9f3b2721ce7b0027" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 33169 } } }
I20250110 20:53:37.147306 5910 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:37.147310 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:37.148109 5914 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:37.153434 5914 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:37.166450 5914 catalog_manager.cc:1348] Generated new cluster ID: c3e895c41ae94b71ad056849684ea6a1
I20250110 20:53:37.166572 5914 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:37.190485 5914 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:53:37.192325 5914 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:37.202780 5914 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a26a7b6e65b7411d9f3b2721ce7b0027: Generated new TSK 0
I20250110 20:53:37.203907 5914 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:53:37.219523 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33169
--builtin_ntp_servers=127.0.104.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250110 20:53:37.546051 5927 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:37.546392 5927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:37.546475 5927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:37.546586 5927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:37.554313 5927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:37.554543 5927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:37.564217 5927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:37.565932 5927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:37.567684 5927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:37.579115 5933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:37.579169 5932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:37.579064 5935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:37.579968 5927 server_base.cc:1034] running on GCE node
I20250110 20:53:37.580529 5927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:37.581502 5927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:37.582692 5927 hybrid_clock.cc:648] HybridClock initialized: now 1736542417582637 us; error 45 us; skew 500 ppm
I20250110 20:53:37.583019 5927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:37.585186 5927 webserver.cc:458] Webserver started at http://127.0.104.1:45687/ using document root <none> and password file <none>
I20250110 20:53:37.585906 5927 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:37.586002 5927 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:37.586248 5927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:37.588781 5927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0de1d30238ce4d10bf85b2377c267e90"
format_stamp: "Formatted at 2025-01-10 20:53:37 on dist-test-slave-1h8z"
I20250110 20:53:37.589474 5927 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0de1d30238ce4d10bf85b2377c267e90"
format_stamp: "Formatted at 2025-01-10 20:53:37 on dist-test-slave-1h8z"
I20250110 20:53:37.594563 5927 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20250110 20:53:37.597913 5941 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:37.599436 5927 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:37.599606 5927 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0de1d30238ce4d10bf85b2377c267e90"
format_stamp: "Formatted at 2025-01-10 20:53:37 on dist-test-slave-1h8z"
I20250110 20:53:37.599807 5927 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:37.621212 5927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:37.622139 5927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:37.622403 5927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:37.623397 5927 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:37.624856 5927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:37.624939 5927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:37.625044 5927 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:37.625116 5927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:37.655537 5927 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:40571
I20250110 20:53:37.655567 6053 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:40571 every 8 connection(s)
I20250110 20:53:37.656894 5927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:37.662814 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 5927
I20250110 20:53:37.663019 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:53:37.666651 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33169
--builtin_ntp_servers=127.0.104.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:53:37.670544 6054 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33169
I20250110 20:53:37.670914 6054 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:37.671917 6054 heartbeater.cc:510] Master 127.0.104.62:33169 requested a full tablet report, sending...
I20250110 20:53:37.674075 5868 ts_manager.cc:194] Registered new tserver with Master: 0de1d30238ce4d10bf85b2377c267e90 (127.0.104.1:40571)
I20250110 20:53:37.675628 5868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:50049
W20250110 20:53:37.986025 6058 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:37.986343 6058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:37.986434 6058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:37.986544 6058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:37.994251 6058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:37.994484 6058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:53:38.003860 6058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:38.005702 6058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:38.007573 6058 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:38.019037 6066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:38.019124 6064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:38.019001 6063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:38.019729 6058 server_base.cc:1034] running on GCE node
I20250110 20:53:38.020282 6058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:38.021181 6058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:38.022385 6058 hybrid_clock.cc:648] HybridClock initialized: now 1736542418022340 us; error 40 us; skew 500 ppm
I20250110 20:53:38.022696 6058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:38.024925 6058 webserver.cc:458] Webserver started at http://127.0.104.2:35567/ using document root <none> and password file <none>
I20250110 20:53:38.025660 6058 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:38.025756 6058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:38.026069 6058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:38.028681 6058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7755aca733b54c59ad51cacb181bb69a"
format_stamp: "Formatted at 2025-01-10 20:53:38 on dist-test-slave-1h8z"
I20250110 20:53:38.029451 6058 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7755aca733b54c59ad51cacb181bb69a"
format_stamp: "Formatted at 2025-01-10 20:53:38 on dist-test-slave-1h8z"
I20250110 20:53:38.034497 6058 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250110 20:53:38.037727 6072 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:38.039165 6058 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:38.039312 6058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7755aca733b54c59ad51cacb181bb69a"
format_stamp: "Formatted at 2025-01-10 20:53:38 on dist-test-slave-1h8z"
I20250110 20:53:38.039494 6058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:38.060132 6058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:38.060971 6058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:38.061272 6058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:38.062237 6058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:38.063711 6058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:38.063812 6058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:38.063916 6058 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:38.063992 6058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:38.093219 6058 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:38313
I20250110 20:53:38.093268 6184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:38313 every 8 connection(s)
I20250110 20:53:38.094488 6058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:53:38.098444 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6058
I20250110 20:53:38.098615 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:53:38.101681 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33169
--builtin_ntp_servers=127.0.104.20:33289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:53:38.107926 6185 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33169
I20250110 20:53:38.108264 6185 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:38.109254 6185 heartbeater.cc:510] Master 127.0.104.62:33169 requested a full tablet report, sending...
I20250110 20:53:38.110822 5868 ts_manager.cc:194] Registered new tserver with Master: 7755aca733b54c59ad51cacb181bb69a (127.0.104.2:38313)
I20250110 20:53:38.111860 5868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:38679
W20250110 20:53:38.428395 6188 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:38.428706 6188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:38.428784 6188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:38.428893 6188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:38.436640 6188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:38.436888 6188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:53:38.446319 6188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:33169
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:38.447883 6188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:38.449542 6188 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:38.460441 6194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:38.460469 6193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:38.461032 6196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:38.462158 6188 server_base.cc:1034] running on GCE node
I20250110 20:53:38.462721 6188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:38.463696 6188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:38.464893 6188 hybrid_clock.cc:648] HybridClock initialized: now 1736542418464835 us; error 44 us; skew 500 ppm
I20250110 20:53:38.465224 6188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:38.471550 6188 webserver.cc:458] Webserver started at http://127.0.104.3:32975/ using document root <none> and password file <none>
I20250110 20:53:38.472361 6188 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:38.472461 6188 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:38.472766 6188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:38.475456 6188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2c5b2eab03f5404da38e53f3125f3cf4"
format_stamp: "Formatted at 2025-01-10 20:53:38 on dist-test-slave-1h8z"
I20250110 20:53:38.476177 6188 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2c5b2eab03f5404da38e53f3125f3cf4"
format_stamp: "Formatted at 2025-01-10 20:53:38 on dist-test-slave-1h8z"
I20250110 20:53:38.481256 6188 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:53:38.484686 6202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:38.486052 6188 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:53:38.486200 6188 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2c5b2eab03f5404da38e53f3125f3cf4"
format_stamp: "Formatted at 2025-01-10 20:53:38 on dist-test-slave-1h8z"
I20250110 20:53:38.486392 6188 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:38.527529 6188 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:38.528404 6188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:38.528676 6188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:38.529665 6188 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:38.531229 6188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:38.531316 6188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:38.531419 6188 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:38.531478 6188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:38.560997 6188 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44755
I20250110 20:53:38.561050 6314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44755 every 8 connection(s)
I20250110 20:53:38.562439 6188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:53:38.566293 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6188
I20250110 20:53:38.566488 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:53:38.574978 6315 heartbeater.cc:346] Connected to a master server at 127.0.104.62:33169
I20250110 20:53:38.575254 6315 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:38.576023 6315 heartbeater.cc:510] Master 127.0.104.62:33169 requested a full tablet report, sending...
I20250110 20:53:38.577338 5868 ts_manager.cc:194] Registered new tserver with Master: 2c5b2eab03f5404da38e53f3125f3cf4 (127.0.104.3:44755)
I20250110 20:53:38.578125 5868 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:40119
I20250110 20:53:38.582213 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:53:38.597146 5868 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59644:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250110 20:53:38.602875 5868 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:53:38.638854 5989 tablet_service.cc:1467] Processing CreateTablet for tablet 308f3c64139c480194e25dceba8cf5f5 (DEFAULT_TABLE table=TestTable [id=ef65a8a9188148beb9a0ead60d4913b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:38.639400 6120 tablet_service.cc:1467] Processing CreateTablet for tablet 308f3c64139c480194e25dceba8cf5f5 (DEFAULT_TABLE table=TestTable [id=ef65a8a9188148beb9a0ead60d4913b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:38.640161 5989 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308f3c64139c480194e25dceba8cf5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:38.640731 6120 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308f3c64139c480194e25dceba8cf5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:38.640383 6250 tablet_service.cc:1467] Processing CreateTablet for tablet 308f3c64139c480194e25dceba8cf5f5 (DEFAULT_TABLE table=TestTable [id=ef65a8a9188148beb9a0ead60d4913b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:38.641922 6250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 308f3c64139c480194e25dceba8cf5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:38.652141 6333 tablet_bootstrap.cc:492] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90: Bootstrap starting.
I20250110 20:53:38.655220 6335 tablet_bootstrap.cc:492] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4: Bootstrap starting.
I20250110 20:53:38.655756 6333 tablet_bootstrap.cc:654] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:38.657202 6333 log.cc:826] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:38.659791 6335 tablet_bootstrap.cc:654] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:38.660444 6334 tablet_bootstrap.cc:492] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a: Bootstrap starting.
I20250110 20:53:38.661403 6335 log.cc:826] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:38.664768 6334 tablet_bootstrap.cc:654] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:38.666028 6334 log.cc:826] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:38.674317 6335 tablet_bootstrap.cc:492] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4: No bootstrap required, opened a new log
I20250110 20:53:38.674610 6335 ts_tablet_manager.cc:1397] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.006s
I20250110 20:53:38.674767 6334 tablet_bootstrap.cc:492] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a: No bootstrap required, opened a new log
I20250110 20:53:38.674768 6333 tablet_bootstrap.cc:492] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90: No bootstrap required, opened a new log
I20250110 20:53:38.675040 6333 ts_tablet_manager.cc:1397] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.014s
I20250110 20:53:38.675040 6334 ts_tablet_manager.cc:1397] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.005s
I20250110 20:53:38.678503 6054 heartbeater.cc:502] Master 127.0.104.62:33169 was elected leader, sending a full tablet report...
I20250110 20:53:38.682703 6335 raft_consensus.cc:357] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.683386 6335 raft_consensus.cc:738] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5b2eab03f5404da38e53f3125f3cf4, State: Initialized, Role: FOLLOWER
I20250110 20:53:38.683462 6333 raft_consensus.cc:357] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.683844 6333 raft_consensus.cc:738] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0de1d30238ce4d10bf85b2377c267e90, State: Initialized, Role: FOLLOWER
I20250110 20:53:38.683970 6335 consensus_queue.cc:260] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.684271 6333 consensus_queue.cc:260] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.684756 6334 raft_consensus.cc:357] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.685190 6334 raft_consensus.cc:738] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7755aca733b54c59ad51cacb181bb69a, State: Initialized, Role: FOLLOWER
I20250110 20:53:38.685674 6334 consensus_queue.cc:260] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.691123 6315 heartbeater.cc:502] Master 127.0.104.62:33169 was elected leader, sending a full tablet report...
I20250110 20:53:38.691277 6335 ts_tablet_manager.cc:1428] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4: Time spent starting tablet: real 0.017s user 0.009s sys 0.009s
I20250110 20:53:38.691717 6333 ts_tablet_manager.cc:1428] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20250110 20:53:38.693776 6185 heartbeater.cc:502] Master 127.0.104.62:33169 was elected leader, sending a full tablet report...
I20250110 20:53:38.694375 6334 ts_tablet_manager.cc:1428] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a: Time spent starting tablet: real 0.019s user 0.010s sys 0.006s
I20250110 20:53:38.713181 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:53:38.715175 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0de1d30238ce4d10bf85b2377c267e90 to finish bootstrapping
I20250110 20:53:38.723229 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7755aca733b54c59ad51cacb181bb69a to finish bootstrapping
I20250110 20:53:38.730695 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2c5b2eab03f5404da38e53f3125f3cf4 to finish bootstrapping
I20250110 20:53:38.744927 6140 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "308f3c64139c480194e25dceba8cf5f5"
dest_uuid: "7755aca733b54c59ad51cacb181bb69a"
from {username='slave'} at 127.0.0.1:42664
I20250110 20:53:38.745218 6140 raft_consensus.cc:491] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:53:38.745384 6140 raft_consensus.cc:3054] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:38.747519 6140 raft_consensus.cc:513] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.748946 6140 leader_election.cc:290] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [CANDIDATE]: Term 1 election: Requested vote from peers 2c5b2eab03f5404da38e53f3125f3cf4 (127.0.104.3:44755), 0de1d30238ce4d10bf85b2377c267e90 (127.0.104.1:40571)
I20250110 20:53:38.757072 6009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308f3c64139c480194e25dceba8cf5f5" candidate_uuid: "7755aca733b54c59ad51cacb181bb69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0de1d30238ce4d10bf85b2377c267e90"
I20250110 20:53:38.757131 6270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "308f3c64139c480194e25dceba8cf5f5" candidate_uuid: "7755aca733b54c59ad51cacb181bb69a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2c5b2eab03f5404da38e53f3125f3cf4"
I20250110 20:53:38.757458 6009 raft_consensus.cc:3054] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:38.757464 6270 raft_consensus.cc:3054] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:38.759555 6009 raft_consensus.cc:2463] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7755aca733b54c59ad51cacb181bb69a in term 1.
I20250110 20:53:38.759563 6270 raft_consensus.cc:2463] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7755aca733b54c59ad51cacb181bb69a in term 1.
I20250110 20:53:38.760166 6074 leader_election.cc:304] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0de1d30238ce4d10bf85b2377c267e90, 7755aca733b54c59ad51cacb181bb69a; no voters:
I20250110 20:53:38.760579 6341 raft_consensus.cc:2798] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:38.761099 6341 raft_consensus.cc:695] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 LEADER]: Becoming Leader. State: Replica: 7755aca733b54c59ad51cacb181bb69a, State: Running, Role: LEADER
I20250110 20:53:38.761564 6341 consensus_queue.cc:237] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.766165 5868 catalog_manager.cc:5526] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7755aca733b54c59ad51cacb181bb69a (127.0.104.2). New cstate: current_term: 1 leader_uuid: "7755aca733b54c59ad51cacb181bb69a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } health_report { overall_health: HEALTHY } } }
I20250110 20:53:38.773411 6270 raft_consensus.cc:1270] T 308f3c64139c480194e25dceba8cf5f5 P 2c5b2eab03f5404da38e53f3125f3cf4 [term 1 FOLLOWER]: Refusing update from remote peer 7755aca733b54c59ad51cacb181bb69a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:38.773411 6009 raft_consensus.cc:1270] T 308f3c64139c480194e25dceba8cf5f5 P 0de1d30238ce4d10bf85b2377c267e90 [term 1 FOLLOWER]: Refusing update from remote peer 7755aca733b54c59ad51cacb181bb69a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:38.774441 6346 consensus_queue.cc:1035] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:38.774713 6341 consensus_queue.cc:1035] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:38.784448 6350 mvcc.cc:204] Tried to move back new op lower bound from 7112877747285413888 to 7112877747251990528. Current Snapshot: MvccSnapshot[applied={T|T < 7112877747285413888}]
I20250110 20:53:38.803745 6140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "308f3c64139c480194e25dceba8cf5f5"
dest_uuid: "7755aca733b54c59ad51cacb181bb69a"
from {username='slave'} at 127.0.0.1:42664
I20250110 20:53:38.804004 6140 raft_consensus.cc:589] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 LEADER]: Received request to step down
I20250110 20:53:38.804091 6140 raft_consensus.cc:3049] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:53:38.804175 6140 raft_consensus.cc:738] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7755aca733b54c59ad51cacb181bb69a, State: Running, Role: LEADER
I20250110 20:53:38.804383 6140 consensus_queue.cc:260] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c5b2eab03f5404da38e53f3125f3cf4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44755 } } peers { permanent_uuid: "0de1d30238ce4d10bf85b2377c267e90" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 40571 } } peers { permanent_uuid: "7755aca733b54c59ad51cacb181bb69a" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 38313 } }
I20250110 20:53:38.804877 6140 raft_consensus.cc:3054] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:38.806896 6140 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "308f3c64139c480194e25dceba8cf5f5"
dest_uuid: "7755aca733b54c59ad51cacb181bb69a"
from {username='slave'} at 127.0.0.1:42664
I20250110 20:53:38.807068 6140 raft_consensus.cc:582] T 308f3c64139c480194e25dceba8cf5f5 P 7755aca733b54c59ad51cacb181bb69a [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20250110 20:53:38.808800 6100 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42664: Illegal state: replica 7755aca733b54c59ad51cacb181bb69a is not leader of this config: current role FOLLOWER
I20250110 20:53:38.810571 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 5927
I20250110 20:53:38.811589 6049 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:38.898557 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 5927
I20250110 20:53:38.912428 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 6058
I20250110 20:53:38.913553 6180 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:39.003242 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6058
I20250110 20:53:39.017959 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 6188
I20250110 20:53:39.019068 6310 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:39.104920 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6188
I20250110 20:53:39.123880 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 5837
I20250110 20:53:39.125092 5898 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:39.227578 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 5837
2025-01-10T20:53:39Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (2603 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20250110 20:53:39.263394 416 test_util.cc:274] Using random seed: 1242144637
I20250110 20:53:39.265132 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:53:39.265219 416 ts_itest-base.cc:113] --------------
I20250110 20:53:39.265281 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:53:39.265342 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:53:39.265390 416 ts_itest-base.cc:116] --------------
2025-01-10T20:53:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:53:39Z Disabled control of system clock
I20250110 20:53:39.294478 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42757
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42757
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250110 20:53:39.614601 6367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:39.615003 6367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:39.615151 6367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:39.622982 6367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:39.623112 6367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:39.623184 6367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:39.623260 6367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:39.632649 6367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42757
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:39.634224 6367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:39.636000 6367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:39.645754 6372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:39.645841 6375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:39.645841 6373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:39.646677 6367 server_base.cc:1034] running on GCE node
I20250110 20:53:39.647459 6367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:39.648850 6367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:39.650070 6367 hybrid_clock.cc:648] HybridClock initialized: now 1736542419650010 us; error 34 us; skew 500 ppm
I20250110 20:53:39.650491 6367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:39.653518 6367 webserver.cc:458] Webserver started at http://127.0.104.62:45771/ using document root <none> and password file <none>
I20250110 20:53:39.654278 6367 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:39.654393 6367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:39.654714 6367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:39.657387 6367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f6dcfb34a0384ff28e203eadb0b9f5be"
format_stamp: "Formatted at 2025-01-10 20:53:39 on dist-test-slave-1h8z"
I20250110 20:53:39.658161 6367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f6dcfb34a0384ff28e203eadb0b9f5be"
format_stamp: "Formatted at 2025-01-10 20:53:39 on dist-test-slave-1h8z"
I20250110 20:53:39.663259 6367 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20250110 20:53:39.666590 6381 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:39.667922 6367 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:39.668087 6367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f6dcfb34a0384ff28e203eadb0b9f5be"
format_stamp: "Formatted at 2025-01-10 20:53:39 on dist-test-slave-1h8z"
I20250110 20:53:39.668260 6367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:39.693526 6367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:39.694406 6367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:39.694690 6367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:39.711252 6367 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42757
I20250110 20:53:39.711290 6432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42757 every 8 connection(s)
I20250110 20:53:39.712590 6367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:39.714372 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6367
I20250110 20:53:39.714571 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:53:39.716724 6433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:39.730028 6433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be: Bootstrap starting.
I20250110 20:53:39.732950 6433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:39.734025 6433 log.cc:826] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:39.736600 6433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be: No bootstrap required, opened a new log
I20250110 20:53:39.742794 6433 raft_consensus.cc:357] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42757 } }
I20250110 20:53:39.743181 6433 raft_consensus.cc:383] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:39.743285 6433 raft_consensus.cc:738] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6dcfb34a0384ff28e203eadb0b9f5be, State: Initialized, Role: FOLLOWER
I20250110 20:53:39.743860 6433 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42757 } }
I20250110 20:53:39.744055 6433 raft_consensus.cc:397] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:39.744175 6433 raft_consensus.cc:491] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:39.744318 6433 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:39.746409 6433 raft_consensus.cc:513] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42757 } }
I20250110 20:53:39.746912 6433 leader_election.cc:304] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6dcfb34a0384ff28e203eadb0b9f5be; no voters:
I20250110 20:53:39.747463 6433 leader_election.cc:290] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:39.747596 6438 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:39.748142 6438 raft_consensus.cc:695] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [term 1 LEADER]: Becoming Leader. State: Replica: f6dcfb34a0384ff28e203eadb0b9f5be, State: Running, Role: LEADER
I20250110 20:53:39.748690 6438 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42757 } }
I20250110 20:53:39.749361 6433 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:39.751242 6440 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6dcfb34a0384ff28e203eadb0b9f5be. Latest consensus state: current_term: 1 leader_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42757 } } }
I20250110 20:53:39.751335 6439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6dcfb34a0384ff28e203eadb0b9f5be" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42757 } } }
I20250110 20:53:39.751588 6440 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:39.751597 6439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:39.753651 6444 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:39.758879 6444 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:39.768401 6444 catalog_manager.cc:1348] Generated new cluster ID: 921e3fd5a8804f61ad62ffd11beb919a
I20250110 20:53:39.768523 6444 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:39.787264 6444 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:53:39.788524 6444 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:39.799960 6444 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f6dcfb34a0384ff28e203eadb0b9f5be: Generated new TSK 0
I20250110 20:53:39.800902 6444 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:53:39.807228 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42757
--builtin_ntp_servers=127.0.104.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20250110 20:53:40.138621 6457 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:40.139019 6457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:40.139092 6457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:40.139170 6457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:40.146950 6457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:40.147145 6457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:40.156726 6457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:40.158218 6457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:40.159976 6457 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:40.171391 6465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:40.171425 6463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:40.171449 6462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:40.172206 6457 server_base.cc:1034] running on GCE node
I20250110 20:53:40.172785 6457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:40.173758 6457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:40.174955 6457 hybrid_clock.cc:648] HybridClock initialized: now 1736542420174903 us; error 47 us; skew 500 ppm
I20250110 20:53:40.175292 6457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:40.177452 6457 webserver.cc:458] Webserver started at http://127.0.104.1:43203/ using document root <none> and password file <none>
I20250110 20:53:40.178198 6457 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:40.178301 6457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:40.178598 6457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:40.181263 6457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "836864310ffd488184d1a393d5545010"
format_stamp: "Formatted at 2025-01-10 20:53:40 on dist-test-slave-1h8z"
I20250110 20:53:40.181978 6457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "836864310ffd488184d1a393d5545010"
format_stamp: "Formatted at 2025-01-10 20:53:40 on dist-test-slave-1h8z"
I20250110 20:53:40.187105 6457 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250110 20:53:40.190372 6471 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:40.191845 6457 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:40.192024 6457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "836864310ffd488184d1a393d5545010"
format_stamp: "Formatted at 2025-01-10 20:53:40 on dist-test-slave-1h8z"
I20250110 20:53:40.192224 6457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:40.206056 6457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:40.206931 6457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:40.207185 6457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:40.208112 6457 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:40.209585 6457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:40.209677 6457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:40.209791 6457 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:40.209859 6457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:40.239250 6457 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33585
I20250110 20:53:40.239305 6583 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33585 every 8 connection(s)
I20250110 20:53:40.240512 6457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:40.241489 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6457
I20250110 20:53:40.241665 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:53:40.245153 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42757
--builtin_ntp_servers=127.0.104.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20250110 20:53:40.253487 6584 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42757
I20250110 20:53:40.253790 6584 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:40.254603 6584 heartbeater.cc:510] Master 127.0.104.62:42757 requested a full tablet report, sending...
I20250110 20:53:40.256906 6398 ts_manager.cc:194] Registered new tserver with Master: 836864310ffd488184d1a393d5545010 (127.0.104.1:33585)
I20250110 20:53:40.258921 6398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:39857
W20250110 20:53:40.568205 6588 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:40.568504 6588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:40.568567 6588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:40.568645 6588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:40.576368 6588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:40.576555 6588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:53:40.585731 6588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:40.587304 6588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:40.588933 6588 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:40.600064 6594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:40.600018 6593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:40.600073 6596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:40.601305 6588 server_base.cc:1034] running on GCE node
I20250110 20:53:40.601935 6588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:40.602933 6588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:40.604130 6588 hybrid_clock.cc:648] HybridClock initialized: now 1736542420604084 us; error 40 us; skew 500 ppm
I20250110 20:53:40.604444 6588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:40.606674 6588 webserver.cc:458] Webserver started at http://127.0.104.2:38975/ using document root <none> and password file <none>
I20250110 20:53:40.607456 6588 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:40.607546 6588 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:40.607853 6588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:40.610424 6588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d0d85bcbac7a4f0f95c986e8970f6267"
format_stamp: "Formatted at 2025-01-10 20:53:40 on dist-test-slave-1h8z"
I20250110 20:53:40.611192 6588 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d0d85bcbac7a4f0f95c986e8970f6267"
format_stamp: "Formatted at 2025-01-10 20:53:40 on dist-test-slave-1h8z"
I20250110 20:53:40.616259 6588 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250110 20:53:40.619598 6602 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:40.621008 6588 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250110 20:53:40.621172 6588 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d0d85bcbac7a4f0f95c986e8970f6267"
format_stamp: "Formatted at 2025-01-10 20:53:40 on dist-test-slave-1h8z"
I20250110 20:53:40.621366 6588 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:40.640234 6588 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:40.641103 6588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:40.641359 6588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:40.642318 6588 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:40.643846 6588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:40.643929 6588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:40.644032 6588 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:40.644101 6588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:40.673660 6588 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33189
I20250110 20:53:40.673738 6714 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33189 every 8 connection(s)
I20250110 20:53:40.675014 6588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:53:40.677775 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6588
I20250110 20:53:40.677960 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:53:40.681270 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42757
--builtin_ntp_servers=127.0.104.20:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20250110 20:53:40.688027 6715 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42757
I20250110 20:53:40.688360 6715 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:40.689184 6715 heartbeater.cc:510] Master 127.0.104.62:42757 requested a full tablet report, sending...
I20250110 20:53:40.690528 6398 ts_manager.cc:194] Registered new tserver with Master: d0d85bcbac7a4f0f95c986e8970f6267 (127.0.104.2:33189)
I20250110 20:53:40.691304 6398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:58213
W20250110 20:53:41.009598 6718 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:41.009930 6718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:41.010025 6718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:41.010138 6718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:41.017899 6718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:41.018131 6718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:53:41.027499 6718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42757
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:41.029049 6718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:41.030730 6718 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:41.041867 6723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:41.041906 6726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:41.041942 6724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:41.042706 6718 server_base.cc:1034] running on GCE node
I20250110 20:53:41.043305 6718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:41.044219 6718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:41.045419 6718 hybrid_clock.cc:648] HybridClock initialized: now 1736542421045369 us; error 52 us; skew 500 ppm
I20250110 20:53:41.045730 6718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:41.047894 6718 webserver.cc:458] Webserver started at http://127.0.104.3:33627/ using document root <none> and password file <none>
I20250110 20:53:41.048611 6718 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:41.048718 6718 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:41.049019 6718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:41.051596 6718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0ddba3aa7a294291881daa084af4d8de"
format_stamp: "Formatted at 2025-01-10 20:53:41 on dist-test-slave-1h8z"
I20250110 20:53:41.052314 6718 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0ddba3aa7a294291881daa084af4d8de"
format_stamp: "Formatted at 2025-01-10 20:53:41 on dist-test-slave-1h8z"
I20250110 20:53:41.057302 6718 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:53:41.060497 6732 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:41.061774 6718 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20250110 20:53:41.061929 6718 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0ddba3aa7a294291881daa084af4d8de"
format_stamp: "Formatted at 2025-01-10 20:53:41 on dist-test-slave-1h8z"
I20250110 20:53:41.062116 6718 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:41.085212 6718 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:41.086055 6718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:41.086308 6718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:41.087309 6718 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:41.088771 6718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:41.088864 6718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:41.088974 6718 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:41.089044 6718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:41.118351 6718 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:39279
I20250110 20:53:41.118400 6844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:39279 every 8 connection(s)
I20250110 20:53:41.119627 6718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:53:41.123525 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6718
I20250110 20:53:41.123685 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:53:41.132203 6845 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42757
I20250110 20:53:41.132512 6845 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:41.133318 6845 heartbeater.cc:510] Master 127.0.104.62:42757 requested a full tablet report, sending...
I20250110 20:53:41.134642 6398 ts_manager.cc:194] Registered new tserver with Master: 0ddba3aa7a294291881daa084af4d8de (127.0.104.3:39279)
I20250110 20:53:41.135440 6398 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:41233
I20250110 20:53:41.139165 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:53:41.153541 6398 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53684:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250110 20:53:41.159111 6398 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:53:41.187531 6650 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5fac80feaf4df69daa65b5de7c2d61 (DEFAULT_TABLE table=TestTable [id=a7c3ce29ab914dfbbf63e7b95ac183f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:41.187453 6519 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5fac80feaf4df69daa65b5de7c2d61 (DEFAULT_TABLE table=TestTable [id=a7c3ce29ab914dfbbf63e7b95ac183f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:41.188900 6519 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c5fac80feaf4df69daa65b5de7c2d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:41.189030 6650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c5fac80feaf4df69daa65b5de7c2d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:41.195053 6780 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5fac80feaf4df69daa65b5de7c2d61 (DEFAULT_TABLE table=TestTable [id=a7c3ce29ab914dfbbf63e7b95ac183f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:41.196563 6780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c5fac80feaf4df69daa65b5de7c2d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:41.199697 6863 tablet_bootstrap.cc:492] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: Bootstrap starting.
I20250110 20:53:41.201320 6864 tablet_bootstrap.cc:492] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: Bootstrap starting.
I20250110 20:53:41.204563 6864 tablet_bootstrap.cc:654] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:41.204563 6863 tablet_bootstrap.cc:654] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:41.205816 6864 log.cc:826] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:41.205816 6863 log.cc:826] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:41.208145 6867 tablet_bootstrap.cc:492] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de: Bootstrap starting.
I20250110 20:53:41.208652 6864 tablet_bootstrap.cc:492] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: No bootstrap required, opened a new log
I20250110 20:53:41.208801 6863 tablet_bootstrap.cc:492] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: No bootstrap required, opened a new log
I20250110 20:53:41.208957 6864 ts_tablet_manager.cc:1397] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250110 20:53:41.209105 6863 ts_tablet_manager.cc:1397] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20250110 20:53:41.211614 6867 tablet_bootstrap.cc:654] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:41.212823 6867 log.cc:826] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:41.215705 6867 tablet_bootstrap.cc:492] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de: No bootstrap required, opened a new log
I20250110 20:53:41.215588 6863 raft_consensus.cc:357] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.215972 6863 raft_consensus.cc:738] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 836864310ffd488184d1a393d5545010, State: Initialized, Role: FOLLOWER
I20250110 20:53:41.216334 6867 ts_tablet_manager.cc:1397] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20250110 20:53:41.216401 6863 consensus_queue.cc:260] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.217494 6584 heartbeater.cc:502] Master 127.0.104.62:42757 was elected leader, sending a full tablet report...
I20250110 20:53:41.217931 6863 ts_tablet_manager.cc:1428] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20250110 20:53:41.217738 6864 raft_consensus.cc:357] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.218152 6864 raft_consensus.cc:738] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0d85bcbac7a4f0f95c986e8970f6267, State: Initialized, Role: FOLLOWER
I20250110 20:53:41.218600 6864 consensus_queue.cc:260] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.219708 6715 heartbeater.cc:502] Master 127.0.104.62:42757 was elected leader, sending a full tablet report...
I20250110 20:53:41.222926 6864 ts_tablet_manager.cc:1428] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: Time spent starting tablet: real 0.014s user 0.011s sys 0.000s
I20250110 20:53:41.225113 6867 raft_consensus.cc:357] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.225569 6867 raft_consensus.cc:738] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddba3aa7a294291881daa084af4d8de, State: Initialized, Role: FOLLOWER
I20250110 20:53:41.226099 6867 consensus_queue.cc:260] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.227311 6845 heartbeater.cc:502] Master 127.0.104.62:42757 was elected leader, sending a full tablet report...
I20250110 20:53:41.227856 6867 ts_tablet_manager.cc:1428] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20250110 20:53:41.239889 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:53:41.242043 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 836864310ffd488184d1a393d5545010 to finish bootstrapping
W20250110 20:53:41.242666 6585 tablet.cc:2367] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:41.251600 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d0d85bcbac7a4f0f95c986e8970f6267 to finish bootstrapping
I20250110 20:53:41.259054 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0ddba3aa7a294291881daa084af4d8de to finish bootstrapping
I20250110 20:53:41.271428 6670 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0c5fac80feaf4df69daa65b5de7c2d61"
dest_uuid: "d0d85bcbac7a4f0f95c986e8970f6267"
from {username='slave'} at 127.0.0.1:51768
I20250110 20:53:41.271731 6670 raft_consensus.cc:491] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:53:41.271883 6670 raft_consensus.cc:3054] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:41.274014 6670 raft_consensus.cc:513] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.275550 6670 leader_election.cc:290] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [CANDIDATE]: Term 1 election: Requested vote from peers 836864310ffd488184d1a393d5545010 (127.0.104.1:33585), 0ddba3aa7a294291881daa084af4d8de (127.0.104.3:39279)
I20250110 20:53:41.282855 6539 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5fac80feaf4df69daa65b5de7c2d61" candidate_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "836864310ffd488184d1a393d5545010"
I20250110 20:53:41.283254 6539 raft_consensus.cc:3054] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:41.283356 6800 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5fac80feaf4df69daa65b5de7c2d61" candidate_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ddba3aa7a294291881daa084af4d8de"
I20250110 20:53:41.283779 6800 raft_consensus.cc:3054] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:41.285604 6539 raft_consensus.cc:2463] T 0c5fac80feaf4df69daa65b5de7c2d61 P 836864310ffd488184d1a393d5545010 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d85bcbac7a4f0f95c986e8970f6267 in term 1.
I20250110 20:53:41.285941 6800 raft_consensus.cc:2463] T 0c5fac80feaf4df69daa65b5de7c2d61 P 0ddba3aa7a294291881daa084af4d8de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0d85bcbac7a4f0f95c986e8970f6267 in term 1.
I20250110 20:53:41.286243 6603 leader_election.cc:304] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 836864310ffd488184d1a393d5545010, d0d85bcbac7a4f0f95c986e8970f6267; no voters:
I20250110 20:53:41.286649 6870 raft_consensus.cc:2798] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:41.287227 6870 raft_consensus.cc:695] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 LEADER]: Becoming Leader. State: Replica: d0d85bcbac7a4f0f95c986e8970f6267, State: Running, Role: LEADER
I20250110 20:53:41.287699 6870 consensus_queue.cc:237] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:41.293416 6398 catalog_manager.cc:5526] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0d85bcbac7a4f0f95c986e8970f6267 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } health_report { overall_health: UNKNOWN } } }
W20250110 20:53:41.426856 6716 tablet.cc:2367] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:42.315050 6670 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0c5fac80feaf4df69daa65b5de7c2d61"
dest_uuid: "d0d85bcbac7a4f0f95c986e8970f6267"
from {username='slave'} at 127.0.0.1:51768
I20250110 20:53:42.315320 6670 raft_consensus.cc:589] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 LEADER]: Received request to step down
I20250110 20:53:42.315423 6670 raft_consensus.cc:3049] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:53:42.315519 6670 raft_consensus.cc:738] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d0d85bcbac7a4f0f95c986e8970f6267, State: Running, Role: LEADER
I20250110 20:53:42.315780 6670 consensus_queue.cc:260] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0d85bcbac7a4f0f95c986e8970f6267" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33189 } } peers { permanent_uuid: "836864310ffd488184d1a393d5545010" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33585 } } peers { permanent_uuid: "0ddba3aa7a294291881daa084af4d8de" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 39279 } }
I20250110 20:53:42.316035 6670 raft_consensus.cc:3054] T 0c5fac80feaf4df69daa65b5de7c2d61 P d0d85bcbac7a4f0f95c986e8970f6267 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:42.317548 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6457
W20250110 20:53:42.330551 6603 connection.cc:537] client connection to 127.0.104.1:33585 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 20:53:42.330894 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 6588
W20250110 20:53:42.330924 6603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 20:53:42.331941 6710 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:42.416909 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6588
I20250110 20:53:42.430488 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6718
I20250110 20:53:42.443203 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 6367
I20250110 20:53:42.444290 6428 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:42.546197 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6367
2025-01-10T20:53:42Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (3318 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20250110 20:53:42.581442 416 test_util.cc:274] Using random seed: 1245462694
I20250110 20:53:42.583297 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:53:42.583365 416 ts_itest-base.cc:113] --------------
I20250110 20:53:42.583412 416 ts_itest-base.cc:114] 5 tablet servers
I20250110 20:53:42.583457 416 ts_itest-base.cc:115] 5 replicas per TS
I20250110 20:53:42.583514 416 ts_itest-base.cc:116] --------------
2025-01-10T20:53:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:53:42Z Disabled control of system clock
I20250110 20:53:42.613417 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42643
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42643
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250110 20:53:42.934319 6890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:42.934638 6890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:42.934799 6890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:42.942631 6890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:42.942775 6890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:42.942860 6890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:42.942939 6890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:42.952376 6890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42643
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:42.953956 6890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:42.955750 6890 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:42.965853 6895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:42.965907 6896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:42.965830 6898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:42.966547 6890 server_base.cc:1034] running on GCE node
I20250110 20:53:42.967325 6890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:42.968631 6890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:42.969843 6890 hybrid_clock.cc:648] HybridClock initialized: now 1736542422969773 us; error 48 us; skew 500 ppm
I20250110 20:53:42.970139 6890 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:42.972273 6890 webserver.cc:458] Webserver started at http://127.0.104.62:46763/ using document root <none> and password file <none>
I20250110 20:53:42.972970 6890 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:42.973066 6890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:42.973332 6890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:42.975948 6890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "fede31d6400f42c78af1200e88a00fa7"
format_stamp: "Formatted at 2025-01-10 20:53:42 on dist-test-slave-1h8z"
I20250110 20:53:42.976697 6890 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "fede31d6400f42c78af1200e88a00fa7"
format_stamp: "Formatted at 2025-01-10 20:53:42 on dist-test-slave-1h8z"
I20250110 20:53:42.981737 6890 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20250110 20:53:42.985057 6904 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:42.986405 6890 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250110 20:53:42.986555 6890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "fede31d6400f42c78af1200e88a00fa7"
format_stamp: "Formatted at 2025-01-10 20:53:42 on dist-test-slave-1h8z"
I20250110 20:53:42.986792 6890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:43.012593 6890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:43.013453 6890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:43.013733 6890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:43.030083 6955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42643 every 8 connection(s)
I20250110 20:53:43.030056 6890 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42643
I20250110 20:53:43.031571 6890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:43.033334 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6890
I20250110 20:53:43.033526 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:53:43.036103 6956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:43.046859 6956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7: Bootstrap starting.
I20250110 20:53:43.049836 6956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:43.050840 6956 log.cc:826] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:43.053503 6956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7: No bootstrap required, opened a new log
I20250110 20:53:43.060004 6956 raft_consensus.cc:357] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fede31d6400f42c78af1200e88a00fa7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42643 } }
I20250110 20:53:43.060381 6956 raft_consensus.cc:383] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:43.060494 6956 raft_consensus.cc:738] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fede31d6400f42c78af1200e88a00fa7, State: Initialized, Role: FOLLOWER
I20250110 20:53:43.061084 6956 consensus_queue.cc:260] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fede31d6400f42c78af1200e88a00fa7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42643 } }
I20250110 20:53:43.061277 6956 raft_consensus.cc:397] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:43.061399 6956 raft_consensus.cc:491] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:43.061544 6956 raft_consensus.cc:3054] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:43.063560 6956 raft_consensus.cc:513] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fede31d6400f42c78af1200e88a00fa7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42643 } }
I20250110 20:53:43.064026 6956 leader_election.cc:304] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fede31d6400f42c78af1200e88a00fa7; no voters:
I20250110 20:53:43.064625 6956 leader_election.cc:290] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:43.064764 6961 raft_consensus.cc:2798] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:43.065541 6961 raft_consensus.cc:695] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [term 1 LEADER]: Becoming Leader. State: Replica: fede31d6400f42c78af1200e88a00fa7, State: Running, Role: LEADER
I20250110 20:53:43.066071 6961 consensus_queue.cc:237] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fede31d6400f42c78af1200e88a00fa7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42643 } }
I20250110 20:53:43.066864 6956 sys_catalog.cc:564] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:43.068979 6962 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fede31d6400f42c78af1200e88a00fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fede31d6400f42c78af1200e88a00fa7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42643 } } }
I20250110 20:53:43.069249 6962 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:43.069574 6963 sys_catalog.cc:455] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fede31d6400f42c78af1200e88a00fa7. Latest consensus state: current_term: 1 leader_uuid: "fede31d6400f42c78af1200e88a00fa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fede31d6400f42c78af1200e88a00fa7" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42643 } } }
I20250110 20:53:43.069813 6963 sys_catalog.cc:458] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7 [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:43.071070 6967 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:43.075382 6967 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:43.084100 6967 catalog_manager.cc:1348] Generated new cluster ID: 0923aaac2bad42e3b37abc6c1b68fef6
I20250110 20:53:43.084234 6967 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:43.124657 6967 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:53:43.126340 6967 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:43.138922 6967 catalog_manager.cc:5899] T 00000000000000000000000000000000 P fede31d6400f42c78af1200e88a00fa7: Generated new TSK 0
I20250110 20:53:43.139976 6967 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:53:43.156944 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--builtin_ntp_servers=127.0.104.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250110 20:53:43.477742 6980 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:43.478088 6980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:43.478179 6980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:43.478291 6980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:43.486311 6980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:43.486523 6980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:43.495904 6980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:43.497460 6980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:43.499166 6980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:43.510475 6988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:43.510491 6985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:43.510496 6986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:43.511396 6980 server_base.cc:1034] running on GCE node
I20250110 20:53:43.511950 6980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:43.512818 6980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:43.514014 6980 hybrid_clock.cc:648] HybridClock initialized: now 1736542423513979 us; error 27 us; skew 500 ppm
I20250110 20:53:43.514294 6980 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:43.520489 6980 webserver.cc:458] Webserver started at http://127.0.104.1:43947/ using document root <none> and password file <none>
I20250110 20:53:43.521243 6980 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:43.521340 6980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:43.521590 6980 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:43.524166 6980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f9f66ee15dfa419ab3706ff4814d1357"
format_stamp: "Formatted at 2025-01-10 20:53:43 on dist-test-slave-1h8z"
I20250110 20:53:43.524860 6980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f9f66ee15dfa419ab3706ff4814d1357"
format_stamp: "Formatted at 2025-01-10 20:53:43 on dist-test-slave-1h8z"
I20250110 20:53:43.529830 6980 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:53:43.533035 6994 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:43.534314 6980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:53:43.534459 6980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f9f66ee15dfa419ab3706ff4814d1357"
format_stamp: "Formatted at 2025-01-10 20:53:43 on dist-test-slave-1h8z"
I20250110 20:53:43.534650 6980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:43.557231 6980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:43.558068 6980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:43.558318 6980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:43.559314 6980 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:43.560755 6980 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:43.560848 6980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:43.560956 6980 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:43.561020 6980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:43.590512 6980 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33165
I20250110 20:53:43.590562 7106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33165 every 8 connection(s)
I20250110 20:53:43.591848 6980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:43.598289 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 6980
I20250110 20:53:43.598485 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:53:43.601617 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--builtin_ntp_servers=127.0.104.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:53:43.604579 7107 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42643
I20250110 20:53:43.604877 7107 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:43.605674 7107 heartbeater.cc:510] Master 127.0.104.62:42643 requested a full tablet report, sending...
I20250110 20:53:43.607964 6921 ts_manager.cc:194] Registered new tserver with Master: f9f66ee15dfa419ab3706ff4814d1357 (127.0.104.1:33165)
I20250110 20:53:43.609496 6921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:55627
W20250110 20:53:43.924288 7111 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:43.924592 7111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:43.924685 7111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:43.924793 7111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:43.932399 7111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:43.932613 7111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:53:43.941911 7111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:43.943502 7111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:43.945165 7111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:43.956669 7116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:43.956758 7117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:43.956732 7119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:43.957512 7111 server_base.cc:1034] running on GCE node
I20250110 20:53:43.958119 7111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:43.959105 7111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:43.960315 7111 hybrid_clock.cc:648] HybridClock initialized: now 1736542423960267 us; error 40 us; skew 500 ppm
I20250110 20:53:43.960621 7111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:43.962807 7111 webserver.cc:458] Webserver started at http://127.0.104.2:39073/ using document root <none> and password file <none>
I20250110 20:53:43.963531 7111 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:43.963639 7111 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:43.963945 7111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:43.966495 7111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ecb8712392eb444699f5911b62c9cdc9"
format_stamp: "Formatted at 2025-01-10 20:53:43 on dist-test-slave-1h8z"
I20250110 20:53:43.967269 7111 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ecb8712392eb444699f5911b62c9cdc9"
format_stamp: "Formatted at 2025-01-10 20:53:43 on dist-test-slave-1h8z"
I20250110 20:53:43.972345 7111 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250110 20:53:43.975629 7125 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:43.976974 7111 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250110 20:53:43.977124 7111 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ecb8712392eb444699f5911b62c9cdc9"
format_stamp: "Formatted at 2025-01-10 20:53:43 on dist-test-slave-1h8z"
I20250110 20:53:43.977311 7111 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:43.990270 7111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:43.991160 7111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:43.991420 7111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:43.992375 7111 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:43.993930 7111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:43.994024 7111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:43.994128 7111 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:43.994197 7111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:44.024339 7111 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41937
I20250110 20:53:44.024391 7237 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41937 every 8 connection(s)
I20250110 20:53:44.025578 7111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:53:44.034341 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7111
I20250110 20:53:44.034510 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:53:44.038326 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--builtin_ntp_servers=127.0.104.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:53:44.039310 7238 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42643
I20250110 20:53:44.039628 7238 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:44.040620 7238 heartbeater.cc:510] Master 127.0.104.62:42643 requested a full tablet report, sending...
I20250110 20:53:44.042321 6921 ts_manager.cc:194] Registered new tserver with Master: ecb8712392eb444699f5911b62c9cdc9 (127.0.104.2:41937)
I20250110 20:53:44.043325 6921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:40601
W20250110 20:53:44.359231 7241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:44.359537 7241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:44.359612 7241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:44.359689 7241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:44.367487 7241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:44.367683 7241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:53:44.376926 7241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:44.378425 7241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:44.380129 7241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:44.391495 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:44.391971 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:44.391999 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:44.392426 7241 server_base.cc:1034] running on GCE node
I20250110 20:53:44.392969 7241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:44.393877 7241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:44.395074 7241 hybrid_clock.cc:648] HybridClock initialized: now 1736542424395026 us; error 43 us; skew 500 ppm
I20250110 20:53:44.395387 7241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:44.397614 7241 webserver.cc:458] Webserver started at http://127.0.104.3:44851/ using document root <none> and password file <none>
I20250110 20:53:44.398351 7241 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:44.398463 7241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:44.398805 7241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:44.401302 7241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b"
format_stamp: "Formatted at 2025-01-10 20:53:44 on dist-test-slave-1h8z"
I20250110 20:53:44.402041 7241 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b"
format_stamp: "Formatted at 2025-01-10 20:53:44 on dist-test-slave-1h8z"
I20250110 20:53:44.407066 7241 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20250110 20:53:44.410316 7255 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:44.411612 7241 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:44.411767 7241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b"
format_stamp: "Formatted at 2025-01-10 20:53:44 on dist-test-slave-1h8z"
I20250110 20:53:44.411947 7241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:44.427060 7241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:44.427875 7241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:44.428123 7241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:44.429075 7241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:44.430524 7241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:44.430626 7241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:44.430727 7241 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:44.430847 7241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250110 20:53:44.459780 7241 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34147
I20250110 20:53:44.459847 7367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34147 every 8 connection(s)
I20250110 20:53:44.461053 7241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:53:44.469579 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7241
I20250110 20:53:44.469736 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:53:44.472831 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.4:0
--local_ip_for_outbound_sockets=127.0.104.4
--webserver_interface=127.0.104.4
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--builtin_ntp_servers=127.0.104.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:53:44.473282 7368 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42643
I20250110 20:53:44.473616 7368 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:44.474602 7368 heartbeater.cc:510] Master 127.0.104.62:42643 requested a full tablet report, sending...
I20250110 20:53:44.476150 6921 ts_manager.cc:194] Registered new tserver with Master: afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3:34147)
I20250110 20:53:44.477113 6921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:49775
I20250110 20:53:44.612425 7107 heartbeater.cc:502] Master 127.0.104.62:42643 was elected leader, sending a full tablet report...
W20250110 20:53:44.794379 7371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:44.794860 7371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:44.794965 7371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:44.795075 7371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:44.802860 7371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:44.803076 7371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.4
I20250110 20:53:44.812314 7371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.0.104.4
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.4
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:44.813836 7371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:44.815536 7371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:44.826835 7376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:44.826846 7377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:44.826880 7379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:44.827348 7371 server_base.cc:1034] running on GCE node
I20250110 20:53:44.828027 7371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:44.828928 7371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:44.830128 7371 hybrid_clock.cc:648] HybridClock initialized: now 1736542424830080 us; error 39 us; skew 500 ppm
I20250110 20:53:44.830437 7371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:44.832608 7371 webserver.cc:458] Webserver started at http://127.0.104.4:45789/ using document root <none> and password file <none>
I20250110 20:53:44.833331 7371 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:44.833431 7371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:44.833752 7371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:44.836273 7371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "1cd93907c84b4e31b468f54e1c0ad817"
format_stamp: "Formatted at 2025-01-10 20:53:44 on dist-test-slave-1h8z"
I20250110 20:53:44.837024 7371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "1cd93907c84b4e31b468f54e1c0ad817"
format_stamp: "Formatted at 2025-01-10 20:53:44 on dist-test-slave-1h8z"
I20250110 20:53:44.842298 7371 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250110 20:53:44.845635 7385 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:44.846984 7371 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:44.847127 7371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "1cd93907c84b4e31b468f54e1c0ad817"
format_stamp: "Formatted at 2025-01-10 20:53:44 on dist-test-slave-1h8z"
I20250110 20:53:44.847309 7371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:44.871356 7371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:44.872200 7371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:44.872452 7371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:44.873363 7371 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:44.874868 7371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:44.874967 7371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:44.875066 7371 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:44.875136 7371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:44.903561 7371 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.4:42405
I20250110 20:53:44.903613 7497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.4:42405 every 8 connection(s)
I20250110 20:53:44.904816 7371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250110 20:53:44.915172 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7371
I20250110 20:53:44.915346 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250110 20:53:44.916781 7498 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42643
I20250110 20:53:44.917090 7498 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:44.918057 7498 heartbeater.cc:510] Master 127.0.104.62:42643 requested a full tablet report, sending...
I20250110 20:53:44.918571 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.5:0
--local_ip_for_outbound_sockets=127.0.104.5
--webserver_interface=127.0.104.5
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--builtin_ntp_servers=127.0.104.20:44713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:53:44.919751 6921 ts_manager.cc:194] Registered new tserver with Master: 1cd93907c84b4e31b468f54e1c0ad817 (127.0.104.4:42405)
I20250110 20:53:44.920475 6921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.4:37279
I20250110 20:53:45.046223 7238 heartbeater.cc:502] Master 127.0.104.62:42643 was elected leader, sending a full tablet report...
W20250110 20:53:45.244661 7501 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:53:45.244997 7501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:45.245095 7501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:45.245209 7501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:45.253023 7501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:45.253238 7501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.5
I20250110 20:53:45.262418 7501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.0.104.5
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42643
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.5
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:45.263984 7501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:45.265640 7501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:45.276715 7509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:45.277115 7507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:45.277163 7506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:45.277552 7501 server_base.cc:1034] running on GCE node
I20250110 20:53:45.278163 7501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:45.279088 7501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:45.280298 7501 hybrid_clock.cc:648] HybridClock initialized: now 1736542425280238 us; error 44 us; skew 500 ppm
I20250110 20:53:45.280627 7501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:45.282922 7501 webserver.cc:458] Webserver started at http://127.0.104.5:44977/ using document root <none> and password file <none>
I20250110 20:53:45.283644 7501 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:45.283744 7501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:45.284046 7501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:45.286609 7501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a57c2f2a7f634d91970a30f566cbf31a"
format_stamp: "Formatted at 2025-01-10 20:53:45 on dist-test-slave-1h8z"
I20250110 20:53:45.287385 7501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a57c2f2a7f634d91970a30f566cbf31a"
format_stamp: "Formatted at 2025-01-10 20:53:45 on dist-test-slave-1h8z"
I20250110 20:53:45.292429 7501 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20250110 20:53:45.295639 7515 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:45.296980 7501 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20250110 20:53:45.297132 7501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a57c2f2a7f634d91970a30f566cbf31a"
format_stamp: "Formatted at 2025-01-10 20:53:45 on dist-test-slave-1h8z"
I20250110 20:53:45.297317 7501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:45.313262 7501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:45.314095 7501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:45.314340 7501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:45.315316 7501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:45.316807 7501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:45.316887 7501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:45.316989 7501 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:45.317057 7501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:45.345824 7501 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.5:44153
I20250110 20:53:45.345863 7627 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.5:44153 every 8 connection(s)
I20250110 20:53:45.347105 7501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250110 20:53:45.351296 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7501
I20250110 20:53:45.351491 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736542259391478-416-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250110 20:53:45.359349 7628 heartbeater.cc:346] Connected to a master server at 127.0.104.62:42643
I20250110 20:53:45.359614 7628 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:45.360344 7628 heartbeater.cc:510] Master 127.0.104.62:42643 requested a full tablet report, sending...
I20250110 20:53:45.361586 6921 ts_manager.cc:194] Registered new tserver with Master: a57c2f2a7f634d91970a30f566cbf31a (127.0.104.5:44153)
I20250110 20:53:45.362285 6921 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.5:57687
I20250110 20:53:45.367197 416 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 20:53:45.381924 6921 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46840:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250110 20:53:45.387533 6921 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20250110 20:53:45.450165 7303 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd60640b904a43a6eefc9781a69480 (DEFAULT_TABLE table=TestTable [id=1b31aa74036741eebc68488ab647db14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:45.450700 7563 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd60640b904a43a6eefc9781a69480 (DEFAULT_TABLE table=TestTable [id=1b31aa74036741eebc68488ab647db14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:45.451706 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78dd60640b904a43a6eefc9781a69480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:45.452127 7563 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78dd60640b904a43a6eefc9781a69480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:45.460022 7042 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd60640b904a43a6eefc9781a69480 (DEFAULT_TABLE table=TestTable [id=1b31aa74036741eebc68488ab647db14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:45.460023 7433 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd60640b904a43a6eefc9781a69480 (DEFAULT_TABLE table=TestTable [id=1b31aa74036741eebc68488ab647db14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:45.461441 7042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78dd60640b904a43a6eefc9781a69480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:45.461992 7433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78dd60640b904a43a6eefc9781a69480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:45.470630 7650 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: Bootstrap starting.
I20250110 20:53:45.474336 7650 tablet_bootstrap.cc:654] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:45.475628 7650 log.cc:826] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:45.479506 7650 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: No bootstrap required, opened a new log
I20250110 20:53:45.479941 7368 heartbeater.cc:502] Master 127.0.104.62:42643 was elected leader, sending a full tablet report...
I20250110 20:53:45.483312 7651 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: Bootstrap starting.
I20250110 20:53:45.483598 7650 ts_tablet_manager.cc:1397] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20250110 20:53:45.483861 7173 tablet_service.cc:1467] Processing CreateTablet for tablet 78dd60640b904a43a6eefc9781a69480 (DEFAULT_TABLE table=TestTable [id=1b31aa74036741eebc68488ab647db14]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:45.485357 7173 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78dd60640b904a43a6eefc9781a69480. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:45.486728 7651 tablet_bootstrap.cc:654] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:45.487977 7651 log.cc:826] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:45.491251 7649 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: Bootstrap starting.
I20250110 20:53:45.492754 7654 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: Bootstrap starting.
I20250110 20:53:45.492668 7650 raft_consensus.cc:357] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.493151 7650 raft_consensus.cc:738] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a57c2f2a7f634d91970a30f566cbf31a, State: Initialized, Role: FOLLOWER
I20250110 20:53:45.493649 7650 consensus_queue.cc:260] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.494962 7649 tablet_bootstrap.cc:654] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:45.495968 7654 tablet_bootstrap.cc:654] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:45.496287 7649 log.cc:826] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:45.497140 7654 log.cc:826] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:45.501609 7658 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: Bootstrap starting.
I20250110 20:53:45.502327 7651 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: No bootstrap required, opened a new log
I20250110 20:53:45.502625 7651 ts_tablet_manager.cc:1397] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20250110 20:53:45.506641 7658 tablet_bootstrap.cc:654] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:45.509299 7649 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: No bootstrap required, opened a new log
I20250110 20:53:45.509670 7649 ts_tablet_manager.cc:1397] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.009s
I20250110 20:53:45.511013 7658 log.cc:826] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:45.511059 7651 raft_consensus.cc:357] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.511518 7651 raft_consensus.cc:738] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd93907c84b4e31b468f54e1c0ad817, State: Initialized, Role: FOLLOWER
I20250110 20:53:45.512049 7651 consensus_queue.cc:260] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.515441 7654 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: No bootstrap required, opened a new log
I20250110 20:53:45.515787 7654 ts_tablet_manager.cc:1397] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20250110 20:53:45.519979 7628 heartbeater.cc:502] Master 127.0.104.62:42643 was elected leader, sending a full tablet report...
I20250110 20:53:45.520763 7650 ts_tablet_manager.cc:1428] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: Time spent starting tablet: real 0.037s user 0.020s sys 0.014s
I20250110 20:53:45.522392 7658 tablet_bootstrap.cc:492] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: No bootstrap required, opened a new log
I20250110 20:53:45.522104 7649 raft_consensus.cc:357] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.522698 7658 ts_tablet_manager.cc:1397] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.006s
I20250110 20:53:45.522706 7649 raft_consensus.cc:738] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afb9dfb8d6d24285b91f70a7d7a0e15b, State: Initialized, Role: FOLLOWER
I20250110 20:53:45.523296 7649 consensus_queue.cc:260] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.527449 7498 heartbeater.cc:502] Master 127.0.104.62:42643 was elected leader, sending a full tablet report...
I20250110 20:53:45.530382 7654 raft_consensus.cc:357] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.530930 7654 raft_consensus.cc:738] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9f66ee15dfa419ab3706ff4814d1357, State: Initialized, Role: FOLLOWER
I20250110 20:53:45.531445 7654 consensus_queue.cc:260] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.531800 7658 raft_consensus.cc:357] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.532284 7658 raft_consensus.cc:738] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb8712392eb444699f5911b62c9cdc9, State: Initialized, Role: FOLLOWER
I20250110 20:53:45.532795 7658 consensus_queue.cc:260] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.534562 7651 ts_tablet_manager.cc:1428] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: Time spent starting tablet: real 0.032s user 0.006s sys 0.012s
I20250110 20:53:45.535946 416 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250110 20:53:45.538826 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f9f66ee15dfa419ab3706ff4814d1357 to finish bootstrapping
I20250110 20:53:45.560374 7658 ts_tablet_manager.cc:1428] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: Time spent starting tablet: real 0.037s user 0.031s sys 0.003s
I20250110 20:53:45.561998 7654 ts_tablet_manager.cc:1428] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: Time spent starting tablet: real 0.046s user 0.018s sys 0.022s
I20250110 20:53:45.564672 7649 ts_tablet_manager.cc:1428] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: Time spent starting tablet: real 0.053s user 0.028s sys 0.019s
I20250110 20:53:45.565181 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ecb8712392eb444699f5911b62c9cdc9 to finish bootstrapping
I20250110 20:53:45.572965 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver afb9dfb8d6d24285b91f70a7d7a0e15b to finish bootstrapping
I20250110 20:53:45.579707 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1cd93907c84b4e31b468f54e1c0ad817 to finish bootstrapping
I20250110 20:53:45.586763 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a57c2f2a7f634d91970a30f566cbf31a to finish bootstrapping
W20250110 20:53:45.594667 7108 tablet.cc:2367] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:45.598551 7629 tablet.cc:2367] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:45.600934 7193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480"
dest_uuid: "ecb8712392eb444699f5911b62c9cdc9"
from {username='slave'} at 127.0.0.1:43806
I20250110 20:53:45.601238 7193 raft_consensus.cc:491] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:53:45.601402 7193 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:45.603555 7193 raft_consensus.cc:513] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.605686 7193 leader_election.cc:290] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [CANDIDATE]: Term 1 election: Requested vote from peers afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3:34147), a57c2f2a7f634d91970a30f566cbf31a (127.0.104.5:44153), 1cd93907c84b4e31b468f54e1c0ad817 (127.0.104.4:42405), f9f66ee15dfa419ab3706ff4814d1357 (127.0.104.1:33165)
I20250110 20:53:45.616744 7453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "ecb8712392eb444699f5911b62c9cdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1cd93907c84b4e31b468f54e1c0ad817"
I20250110 20:53:45.616729 7062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "ecb8712392eb444699f5911b62c9cdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f9f66ee15dfa419ab3706ff4814d1357"
I20250110 20:53:45.616995 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "ecb8712392eb444699f5911b62c9cdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a57c2f2a7f634d91970a30f566cbf31a"
I20250110 20:53:45.617152 7453 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:45.617154 7062 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:45.617370 7583 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:45.619340 7453 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb8712392eb444699f5911b62c9cdc9 in term 1.
I20250110 20:53:45.620195 7062 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb8712392eb444699f5911b62c9cdc9 in term 1.
I20250110 20:53:45.620234 7583 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb8712392eb444699f5911b62c9cdc9 in term 1.
I20250110 20:53:45.620869 7126 leader_election.cc:304] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 1cd93907c84b4e31b468f54e1c0ad817, a57c2f2a7f634d91970a30f566cbf31a, ecb8712392eb444699f5911b62c9cdc9; no voters:
I20250110 20:53:45.620990 7323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "ecb8712392eb444699f5911b62c9cdc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b"
I20250110 20:53:45.621328 7663 raft_consensus.cc:2798] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:45.621381 7323 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:45.621846 7663 raft_consensus.cc:695] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 LEADER]: Becoming Leader. State: Replica: ecb8712392eb444699f5911b62c9cdc9, State: Running, Role: LEADER
I20250110 20:53:45.622404 7663 consensus_queue.cc:237] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:45.623579 7323 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb8712392eb444699f5911b62c9cdc9 in term 1.
I20250110 20:53:45.627382 6919 catalog_manager.cc:5526] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecb8712392eb444699f5911b62c9cdc9 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "ecb8712392eb444699f5911b62c9cdc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } health_report { overall_health: UNKNOWN } } }
W20250110 20:53:45.656685 7499 tablet.cc:2367] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:45.713204 7369 tablet.cc:2367] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:45.779034 7239 tablet.cc:2367] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:46.168232 7672 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:46.173751 7672 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:46.186679 7679 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:46.190845 7677 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:46.193516 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 6890
I20250110 20:53:46.194555 6951 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:46.314986 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6890
W20250110 20:53:46.332707 7628 heartbeater.cc:643] Failed to heartbeat to 127.0.104.62:42643 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.62:42643 (error 108)
W20250110 20:53:46.332870 7107 heartbeater.cc:643] Failed to heartbeat to 127.0.104.62:42643 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.62:42643 (error 108)
I20250110 20:53:46.333041 416 raft_consensus_election-itest.cc:666] Removing tserver with uuid afb9dfb8d6d24285b91f70a7d7a0e15b
I20250110 20:53:46.335305 7193 consensus_queue.cc:237] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }
I20250110 20:53:46.337249 7062 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:46.337253 7453 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:46.337657 7583 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:53:46.337893 7681 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:53:46.338239 7679 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:53:46.338640 7677 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:53:46.341257 7681 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } }
I20250110 20:53:46.341929 7453 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } }
I20250110 20:53:46.341991 7583 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } }
I20250110 20:53:46.345399 7062 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } }
W20250110 20:53:46.347225 7238 heartbeater.cc:643] Failed to heartbeat to 127.0.104.62:42643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:42643: connect: Connection refused (error 111)
W20250110 20:53:46.348031 7498 heartbeater.cc:643] Failed to heartbeat to 127.0.104.62:42643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:42643: connect: Connection refused (error 111)
I20250110 20:53:46.377671 416 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20250110 20:53:46.410322 416 raft_consensus_election-itest.cc:684] Adding back Peer afb9dfb8d6d24285b91f70a7d7a0e15b and expecting timeout...
I20250110 20:53:46.412086 7193 consensus_queue.cc:237] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } }
I20250110 20:53:46.414213 7323 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 20:53:46.414716 7681 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:53:46.417021 7323 raft_consensus.cc:867] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } }
W20250110 20:53:46.417141 7323 raft_consensus.cc:1574] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } } peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } }
I20250110 20:53:46.417666 7323 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } }
I20250110 20:53:46.421720 7323 raft_consensus.cc:1476] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
W20250110 20:53:46.422106 7368 heartbeater.cc:643] Failed to heartbeat to 127.0.104.62:42643 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:42643: connect: Connection refused (error 111)
I20250110 20:53:46.511687 416 raft_consensus_election-itest.cc:688] Timeout achieved.
I20250110 20:53:46.519229 416 raft_consensus_election-itest.cc:699] Forcing Peer ecb8712392eb444699f5911b62c9cdc9 to step down...
I20250110 20:53:46.519891 7193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480"
dest_uuid: "ecb8712392eb444699f5911b62c9cdc9"
from {username='slave'} at 127.0.0.1:43806
I20250110 20:53:46.520133 7193 raft_consensus.cc:589] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 LEADER]: Received request to step down
I20250110 20:53:46.520208 7193 raft_consensus.cc:3049] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:53:46.520294 7193 raft_consensus.cc:738] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ecb8712392eb444699f5911b62c9cdc9, State: Running, Role: LEADER
I20250110 20:53:46.520556 7193 consensus_queue.cc:260] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } }
I20250110 20:53:46.520895 7193 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:46.521400 416 raft_consensus_election-itest.cc:705] Resuming Peer f9f66ee15dfa419ab3706ff4814d1357 ...
I20250110 20:53:46.522037 416 raft_consensus_election-itest.cc:710] Trying to elect Peer afb9dfb8d6d24285b91f70a7d7a0e15b ...
I20250110 20:53:46.522622 7062 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 20:53:46.523070 7323 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480"
dest_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b"
from {username='slave'} at 127.0.0.1:59148
I20250110 20:53:46.523313 7323 raft_consensus.cc:491] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:53:46.523437 7323 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:46.525300 7323 raft_consensus.cc:513] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } }
I20250110 20:53:46.526993 7323 leader_election.cc:290] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [CANDIDATE]: Term 2 election: Requested vote from peers ecb8712392eb444699f5911b62c9cdc9 (127.0.104.2:41937), a57c2f2a7f634d91970a30f566cbf31a (127.0.104.5:44153), 1cd93907c84b4e31b468f54e1c0ad817 (127.0.104.4:42405), f9f66ee15dfa419ab3706ff4814d1357 (127.0.104.1:33165)
I20250110 20:53:46.532120 416 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20250110 20:53:46.534775 7062 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f9f66ee15dfa419ab3706ff4814d1357"
I20250110 20:53:46.535003 7062 raft_consensus.cc:1808] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 1 FOLLOWER]: Handling vote request from an unknown peer afb9dfb8d6d24285b91f70a7d7a0e15b
I20250110 20:53:46.535135 7062 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:46.535768 7193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "ecb8712392eb444699f5911b62c9cdc9"
I20250110 20:53:46.537761 7062 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb9dfb8d6d24285b91f70a7d7a0e15b in term 2.
I20250110 20:53:46.538419 7193 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb9dfb8d6d24285b91f70a7d7a0e15b in term 2.
I20250110 20:53:46.538878 7257 leader_election.cc:304] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: afb9dfb8d6d24285b91f70a7d7a0e15b, ecb8712392eb444699f5911b62c9cdc9, f9f66ee15dfa419ab3706ff4814d1357; no voters:
I20250110 20:53:46.539217 7675 raft_consensus.cc:2798] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:53:46.539705 7675 raft_consensus.cc:695] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 2 LEADER]: Becoming Leader. State: Replica: afb9dfb8d6d24285b91f70a7d7a0e15b, State: Running, Role: LEADER
I20250110 20:53:46.540257 7675 consensus_queue.cc:237] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } }
I20250110 20:53:46.635473 416 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20250110 20:53:46.838712 416 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20250110 20:53:47.033819 7062 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 2 FOLLOWER]: Refusing update from remote peer afb9dfb8d6d24285b91f70a7d7a0e15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 20:53:47.034497 7675 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 20:53:47.141886 416 cluster_itest_util.cc:256] Not converged past 4 yet: 2.4 1.3 2.4
I20250110 20:53:47.160490 7193 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 2 FOLLOWER]: Refusing update from remote peer afb9dfb8d6d24285b91f70a7d7a0e15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 20:53:47.161178 7675 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 20:53:47.164073 7709 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } } }
I20250110 20:53:47.164572 7193 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P ecb8712392eb444699f5911b62c9cdc9 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } } }
W20250110 20:53:47.167135 7193 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43806 (request call id 56) took 755 ms (client timeout 100 ms). Trace:
W20250110 20:53:47.167279 7193 rpcz_store.cc:269] 0110 20:53:46.411184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:46.411278 (+ 94us) service_pool.cc:225] Handling call
0110 20:53:46.413020 (+ 1742us) log.cc:844] Serialized 591 byte log entry
0110 20:53:47.167070 (+754050us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":260,"raft.queue_time_us":632,"raft.run_cpu_time_us":750,"raft.run_wall_time_us":854,"spinlock_wait_cycles":311936}
I20250110 20:53:47.169178 7062 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P f9f66ee15dfa419ab3706ff4814d1357 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } } }
I20250110 20:53:47.545248 416 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20250110 20:53:47.547456 7583 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 20:53:47.550001 7453 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 1 FOLLOWER]: Refusing update from remote peer ecb8712392eb444699f5911b62c9cdc9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250110 20:53:47.563465 7452 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:47.564787 7453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "1cd93907c84b4e31b468f54e1c0ad817"
I20250110 20:53:47.566443 7452 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 2 FOLLOWER]: Refusing update from remote peer afb9dfb8d6d24285b91f70a7d7a0e15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 20:53:47.566982 7453 raft_consensus.cc:1808] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 2 FOLLOWER]: Handling vote request from an unknown peer afb9dfb8d6d24285b91f70a7d7a0e15b
I20250110 20:53:47.567130 7583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78dd60640b904a43a6eefc9781a69480" candidate_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "a57c2f2a7f634d91970a30f566cbf31a"
I20250110 20:53:47.567175 7712 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 20:53:47.567317 7583 raft_consensus.cc:1808] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 1 FOLLOWER]: Handling vote request from an unknown peer afb9dfb8d6d24285b91f70a7d7a0e15b
I20250110 20:53:47.567412 7583 raft_consensus.cc:3054] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:47.569842 7583 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb9dfb8d6d24285b91f70a7d7a0e15b in term 2.
I20250110 20:53:47.569932 7453 raft_consensus.cc:2463] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afb9dfb8d6d24285b91f70a7d7a0e15b in term 2.
I20250110 20:53:47.570334 7582 raft_consensus.cc:1270] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 2 FOLLOWER]: Refusing update from remote peer afb9dfb8d6d24285b91f70a7d7a0e15b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250110 20:53:47.571864 7709 consensus_queue.cc:1035] T 78dd60640b904a43a6eefc9781a69480 P afb9dfb8d6d24285b91f70a7d7a0e15b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250110 20:53:47.587071 7452 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P 1cd93907c84b4e31b468f54e1c0ad817 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } } }
I20250110 20:53:47.587952 7582 raft_consensus.cc:2949] T 78dd60640b904a43a6eefc9781a69480 P a57c2f2a7f634d91970a30f566cbf31a [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER afb9dfb8d6d24285b91f70a7d7a0e15b (127.0.104.3) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ecb8712392eb444699f5911b62c9cdc9" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41937 } } peers { permanent_uuid: "a57c2f2a7f634d91970a30f566cbf31a" member_type: VOTER last_known_addr { host: "127.0.104.5" port: 44153 } } peers { permanent_uuid: "1cd93907c84b4e31b468f54e1c0ad817" member_type: VOTER last_known_addr { host: "127.0.104.4" port: 42405 } } peers { permanent_uuid: "f9f66ee15dfa419ab3706ff4814d1357" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33165 } } peers { permanent_uuid: "afb9dfb8d6d24285b91f70a7d7a0e15b" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34147 } attrs { } } }
I20250110 20:53:47.596412 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 6980
I20250110 20:53:47.597569 7102 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:47.684027 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 6980
I20250110 20:53:47.697947 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 7111
I20250110 20:53:47.699000 7233 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:47.788060 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 7111
I20250110 20:53:47.802445 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 7241
I20250110 20:53:47.803481 7363 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:47.892242 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 7241
I20250110 20:53:47.905728 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 7371
I20250110 20:53:47.906766 7493 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:47.993477 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 7371
I20250110 20:53:48.006557 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 7501
I20250110 20:53:48.007671 7623 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:48.093649 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 7501
2025-01-10T20:53:48Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (5545 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20250110 20:53:48.127205 416 test_util.cc:274] Using random seed: 1251008449
I20250110 20:53:48.128958 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:53:48.129040 416 ts_itest-base.cc:113] --------------
I20250110 20:53:48.129088 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:53:48.129150 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:53:48.129196 416 ts_itest-base.cc:116] --------------
2025-01-10T20:53:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:53:48Z Disabled control of system clock
I20250110 20:53:48.158232 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44373
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44373
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20250110 20:53:48.471774 7741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:48.472095 7741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:48.472223 7741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:48.479809 7741 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:48.479979 7741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:48.480042 7741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:48.480123 7741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:48.480188 7741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:48.489426 7741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44373
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:48.490963 7741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:48.492604 7741 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:48.502214 7749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:48.502156 7747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:48.502492 7746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:48.503065 7741 server_base.cc:1034] running on GCE node
I20250110 20:53:48.503795 7741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:48.505093 7741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:48.506281 7741 hybrid_clock.cc:648] HybridClock initialized: now 1736542428506239 us; error 36 us; skew 500 ppm
I20250110 20:53:48.506585 7741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:48.508677 7741 webserver.cc:458] Webserver started at http://127.0.104.62:36231/ using document root <none> and password file <none>
I20250110 20:53:48.509363 7741 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:48.509461 7741 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:48.509758 7741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:48.512245 7741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1a4f2670dfce41239d1db9bfeef9d76f"
format_stamp: "Formatted at 2025-01-10 20:53:48 on dist-test-slave-1h8z"
I20250110 20:53:48.512981 7741 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1a4f2670dfce41239d1db9bfeef9d76f"
format_stamp: "Formatted at 2025-01-10 20:53:48 on dist-test-slave-1h8z"
I20250110 20:53:48.517870 7741 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250110 20:53:48.521095 7755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:48.522352 7741 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:53:48.522500 7741 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1a4f2670dfce41239d1db9bfeef9d76f"
format_stamp: "Formatted at 2025-01-10 20:53:48 on dist-test-slave-1h8z"
I20250110 20:53:48.522703 7741 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:48.543450 7741 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:48.544287 7741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:48.544544 7741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:48.560927 7741 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44373
I20250110 20:53:48.560953 7806 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44373 every 8 connection(s)
I20250110 20:53:48.562238 7741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:48.565883 7807 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:48.566923 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7741
I20250110 20:53:48.567103 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:53:48.576292 7807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Bootstrap starting.
I20250110 20:53:48.579914 7807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:48.581017 7807 log.cc:826] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:48.583560 7807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: No bootstrap required, opened a new log
I20250110 20:53:48.589740 7807 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:48.590078 7807 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:48.590157 7807 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4f2670dfce41239d1db9bfeef9d76f, State: Initialized, Role: FOLLOWER
I20250110 20:53:48.590677 7807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:48.590911 7807 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:48.591008 7807 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:48.591128 7807 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:48.593230 7807 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:48.593675 7807 leader_election.cc:304] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a4f2670dfce41239d1db9bfeef9d76f; no voters:
I20250110 20:53:48.594208 7807 leader_election.cc:290] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:53:48.594347 7812 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:48.594913 7812 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 LEADER]: Becoming Leader. State: Replica: 1a4f2670dfce41239d1db9bfeef9d76f, State: Running, Role: LEADER
I20250110 20:53:48.595427 7812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:48.596125 7807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:48.597986 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } } }
I20250110 20:53:48.598238 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:48.600073 7814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4f2670dfce41239d1db9bfeef9d76f. Latest consensus state: current_term: 1 leader_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } } }
I20250110 20:53:48.600395 7814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:48.600945 7818 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:48.604889 7818 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:48.612903 7818 catalog_manager.cc:1348] Generated new cluster ID: dccc8f2f36a84c04bdec375a22cd8ad6
I20250110 20:53:48.613025 7818 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:48.621507 7818 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:53:48.622689 7818 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:48.640478 7818 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Generated new TSK 0
I20250110 20:53:48.641295 7818 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:53:48.647681 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44373
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20250110 20:53:48.972532 7831 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250110 20:53:48.972889 7831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:48.972978 7831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:48.973093 7831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:48.980873 7831 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250110 20:53:48.980968 7831 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250110 20:53:48.981035 7831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:48.981122 7831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:48.981280 7831 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250110 20:53:48.981343 7831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:53:48.990764 7831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:48.992326 7831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:48.994179 7831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:49.005621 7839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:49.005649 7837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:49.005621 7836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:49.006479 7831 server_base.cc:1034] running on GCE node
I20250110 20:53:49.007097 7831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:49.008009 7831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:49.009208 7831 hybrid_clock.cc:648] HybridClock initialized: now 1736542429009165 us; error 39 us; skew 500 ppm
I20250110 20:53:49.009518 7831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:49.011698 7831 webserver.cc:458] Webserver started at http://127.0.104.1:42625/ using document root <none> and password file <none>
I20250110 20:53:49.012426 7831 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:49.012524 7831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:49.012843 7831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:49.015448 7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "deae654cdde442b39e56ee12665b00b5"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.016172 7831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "deae654cdde442b39e56ee12665b00b5"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.021324 7831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250110 20:53:49.024582 7845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.025975 7831 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:53:49.026140 7831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "deae654cdde442b39e56ee12665b00b5"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.026325 7831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:49.047036 7831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:49.047914 7831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:49.048173 7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:49.049101 7831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:49.050537 7831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:49.050633 7831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.050765 7831 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:49.050841 7831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250110 20:53:49.079414 7831 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:44337
I20250110 20:53:49.079457 7957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:44337 every 8 connection(s)
I20250110 20:53:49.080651 7831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:53:49.090586 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7831
I20250110 20:53:49.090819 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:53:49.092813 7958 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44373
I20250110 20:53:49.093086 7958 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:49.093844 7958 heartbeater.cc:510] Master 127.0.104.62:44373 requested a full tablet report, sending...
I20250110 20:53:49.094275 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44373
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250110 20:53:49.095722 7771 ts_manager.cc:194] Registered new tserver with Master: deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337)
I20250110 20:53:49.097249 7771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:56819
W20250110 20:53:49.409644 7962 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250110 20:53:49.409965 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:49.410027 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:49.410102 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:49.417721 7962 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250110 20:53:49.417812 7962 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250110 20:53:49.417877 7962 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:49.417934 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:49.418051 7962 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250110 20:53:49.418105 7962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:53:49.427371 7962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:49.428884 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:49.430474 7962 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:49.441381 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:49.441438 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:49.441465 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:49.442322 7962 server_base.cc:1034] running on GCE node
I20250110 20:53:49.442952 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:49.443864 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:49.445062 7962 hybrid_clock.cc:648] HybridClock initialized: now 1736542429445011 us; error 39 us; skew 500 ppm
I20250110 20:53:49.445341 7962 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:49.447427 7962 webserver.cc:458] Webserver started at http://127.0.104.2:41429/ using document root <none> and password file <none>
I20250110 20:53:49.448137 7962 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:49.448235 7962 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:49.448535 7962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:49.451066 7962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0095250bbe784e2da1ba34c4537900af"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.451767 7962 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0095250bbe784e2da1ba34c4537900af"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.456543 7962 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20250110 20:53:49.459610 7976 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.460871 7962 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250110 20:53:49.461026 7962 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0095250bbe784e2da1ba34c4537900af"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.461203 7962 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:49.483170 7962 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:49.483995 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:49.484251 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:49.485203 7962 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:49.486644 7962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:49.486765 7962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.486883 7962 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:49.486953 7962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.515648 7962 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43797
I20250110 20:53:49.515694 8088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43797 every 8 connection(s)
I20250110 20:53:49.516934 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:53:49.524787 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 7962
I20250110 20:53:49.524994 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:53:49.528425 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44373
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250110 20:53:49.529330 8089 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44373
I20250110 20:53:49.529606 8089 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:49.530285 8089 heartbeater.cc:510] Master 127.0.104.62:44373 requested a full tablet report, sending...
I20250110 20:53:49.531594 7771 ts_manager.cc:194] Registered new tserver with Master: 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
I20250110 20:53:49.532331 7771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:34345
W20250110 20:53:49.844883 8092 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250110 20:53:49.845208 8092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:49.845273 8092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:49.845346 8092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:49.852891 8092 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250110 20:53:49.852980 8092 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250110 20:53:49.853044 8092 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:49.853099 8092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:49.853215 8092 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250110 20:53:49.853267 8092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:53:49.862337 8092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:49.863839 8092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:49.865377 8092 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:49.876016 8097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:49.876062 8098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:49.876065 8100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:49.877055 8092 server_base.cc:1034] running on GCE node
I20250110 20:53:49.877597 8092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:49.878463 8092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:49.879648 8092 hybrid_clock.cc:648] HybridClock initialized: now 1736542429879591 us; error 52 us; skew 500 ppm
I20250110 20:53:49.879941 8092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:49.881947 8092 webserver.cc:458] Webserver started at http://127.0.104.3:37277/ using document root <none> and password file <none>
I20250110 20:53:49.882650 8092 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:49.882772 8092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:49.883069 8092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:53:49.885526 8092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "18c6b6ae45d6423c9b540c94c2266b3f"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.886209 8092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "18c6b6ae45d6423c9b540c94c2266b3f"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.891033 8092 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.005s sys 0.003s
I20250110 20:53:49.894025 8106 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.895232 8092 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:53:49.895377 8092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "18c6b6ae45d6423c9b540c94c2266b3f"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:49.895555 8092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:49.941988 8092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:49.943065 8092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:49.943334 8092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:49.944429 8092 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:49.946106 8092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:53:49.946202 8092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.946307 8092 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:49.946379 8092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:49.975093 8092 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42561
I20250110 20:53:49.975149 8218 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42561 every 8 connection(s)
I20250110 20:53:49.976318 8092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:53:49.980711 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8092
I20250110 20:53:49.980892 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:53:49.988497 8219 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44373
I20250110 20:53:49.988754 8219 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:49.989385 8219 heartbeater.cc:510] Master 127.0.104.62:44373 requested a full tablet report, sending...
I20250110 20:53:49.990561 7771 ts_manager.cc:194] Registered new tserver with Master: 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561)
I20250110 20:53:49.991308 7771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:45459
I20250110 20:53:49.996161 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:53:50.010111 7771 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39058:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250110 20:53:50.039038 7893 tablet_service.cc:1467] Processing CreateTablet for tablet b287ad5536de4f10929a2dbe394a3bc3 (DEFAULT_TABLE table=TestTable [id=9bc3bedc4316457eb9bf9e6bc6f26aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:50.039887 8154 tablet_service.cc:1467] Processing CreateTablet for tablet b287ad5536de4f10929a2dbe394a3bc3 (DEFAULT_TABLE table=TestTable [id=9bc3bedc4316457eb9bf9e6bc6f26aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:50.040434 7893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b287ad5536de4f10929a2dbe394a3bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:50.041111 8154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b287ad5536de4f10929a2dbe394a3bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:50.042773 8024 tablet_service.cc:1467] Processing CreateTablet for tablet b287ad5536de4f10929a2dbe394a3bc3 (DEFAULT_TABLE table=TestTable [id=9bc3bedc4316457eb9bf9e6bc6f26aab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:53:50.044067 8024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b287ad5536de4f10929a2dbe394a3bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:50.051141 8237 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Bootstrap starting.
I20250110 20:53:50.053063 8238 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Bootstrap starting.
I20250110 20:53:50.054512 8237 tablet_bootstrap.cc:654] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:50.055783 8237 log.cc:826] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:50.056512 8238 tablet_bootstrap.cc:654] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:50.057614 8239 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Bootstrap starting.
I20250110 20:53:50.057880 8238 log.cc:826] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:50.060302 8238 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: No bootstrap required, opened a new log
I20250110 20:53:50.060549 8238 ts_tablet_manager.cc:1397] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20250110 20:53:50.060760 8237 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: No bootstrap required, opened a new log
I20250110 20:53:50.060843 8239 tablet_bootstrap.cc:654] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Neither blocks nor log segments found. Creating new log.
I20250110 20:53:50.061020 8237 ts_tablet_manager.cc:1397] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20250110 20:53:50.061964 8239 log.cc:826] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:50.064426 8239 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: No bootstrap required, opened a new log
I20250110 20:53:50.064960 8239 ts_tablet_manager.cc:1397] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20250110 20:53:50.067008 8238 raft_consensus.cc:357] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.067341 8238 raft_consensus.cc:383] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:50.067481 8238 raft_consensus.cc:738] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deae654cdde442b39e56ee12665b00b5, State: Initialized, Role: FOLLOWER
I20250110 20:53:50.068048 8238 consensus_queue.cc:260] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.069056 7958 heartbeater.cc:502] Master 127.0.104.62:44373 was elected leader, sending a full tablet report...
I20250110 20:53:50.069183 8237 raft_consensus.cc:357] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.069571 8237 raft_consensus.cc:383] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:50.069658 8237 raft_consensus.cc:738] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6b6ae45d6423c9b540c94c2266b3f, State: Initialized, Role: FOLLOWER
I20250110 20:53:50.069667 8238 ts_tablet_manager.cc:1428] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20250110 20:53:50.070312 8237 consensus_queue.cc:260] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.071359 8219 heartbeater.cc:502] Master 127.0.104.62:44373 was elected leader, sending a full tablet report...
I20250110 20:53:50.071841 8237 ts_tablet_manager.cc:1428] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20250110 20:53:50.073841 8239 raft_consensus.cc:357] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.074180 8239 raft_consensus.cc:383] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:53:50.074270 8239 raft_consensus.cc:738] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0095250bbe784e2da1ba34c4537900af, State: Initialized, Role: FOLLOWER
I20250110 20:53:50.074790 8239 consensus_queue.cc:260] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.075842 8089 heartbeater.cc:502] Master 127.0.104.62:44373 was elected leader, sending a full tablet report...
I20250110 20:53:50.076236 8239 ts_tablet_manager.cc:1428] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20250110 20:53:50.077792 8220 tablet.cc:2367] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:50.083680 7959 tablet.cc:2367] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:53:50.119421 8090 tablet.cc:2367] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:53:50.174257 8245 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:50.174516 8245 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.176203 8245 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337), 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561)
I20250110 20:53:50.183301 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "0095250bbe784e2da1ba34c4537900af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deae654cdde442b39e56ee12665b00b5" is_pre_election: true
I20250110 20:53:50.183501 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "0095250bbe784e2da1ba34c4537900af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" is_pre_election: true
I20250110 20:53:50.183794 7913 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0095250bbe784e2da1ba34c4537900af in term 0.
I20250110 20:53:50.183952 8174 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0095250bbe784e2da1ba34c4537900af in term 0.
I20250110 20:53:50.184391 7980 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0095250bbe784e2da1ba34c4537900af, deae654cdde442b39e56ee12665b00b5; no voters:
I20250110 20:53:50.184741 8245 raft_consensus.cc:2798] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:53:50.184867 8245 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:50.184957 8245 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:50.186902 8245 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.187423 8245 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [CANDIDATE]: Term 1 election: Requested vote from peers deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337), 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561)
I20250110 20:53:50.187877 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "0095250bbe784e2da1ba34c4537900af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deae654cdde442b39e56ee12665b00b5"
I20250110 20:53:50.187937 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "0095250bbe784e2da1ba34c4537900af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "18c6b6ae45d6423c9b540c94c2266b3f"
I20250110 20:53:50.188035 7913 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:50.188105 8174 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:53:50.189981 8174 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0095250bbe784e2da1ba34c4537900af in term 1.
I20250110 20:53:50.189999 7913 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0095250bbe784e2da1ba34c4537900af in term 1.
I20250110 20:53:50.190320 7980 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0095250bbe784e2da1ba34c4537900af, deae654cdde442b39e56ee12665b00b5; no voters:
I20250110 20:53:50.190593 8245 raft_consensus.cc:2798] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:53:50.191011 8245 raft_consensus.cc:695] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 1 LEADER]: Becoming Leader. State: Replica: 0095250bbe784e2da1ba34c4537900af, State: Running, Role: LEADER
I20250110 20:53:50.191426 8245 consensus_queue.cc:237] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:50.195554 7772 catalog_manager.cc:5526] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af reported cstate change: term changed from 0 to 1, leader changed from <none> to 0095250bbe784e2da1ba34c4537900af (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0095250bbe784e2da1ba34c4537900af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } } }
I20250110 20:53:50.220939 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:53:50.222925 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver deae654cdde442b39e56ee12665b00b5 to finish bootstrapping
I20250110 20:53:50.230597 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0095250bbe784e2da1ba34c4537900af to finish bootstrapping
I20250110 20:53:50.237233 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 18c6b6ae45d6423c9b540c94c2266b3f to finish bootstrapping
I20250110 20:53:50.259819 416 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0 0.0
I20250110 20:53:50.363086 416 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0 0.0
I20250110 20:53:50.566130 416 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0 0.0
I20250110 20:53:50.654524 8245 consensus_queue.cc:1035] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [LEADER]: Connected to new peer: Peer: permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:50.660959 8245 consensus_queue.cc:1035] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [LEADER]: Connected to new peer: Peer: permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:53:50.982551 416 test_util.cc:274] Using random seed: 1253863790
I20250110 20:53:51.288997 416 test_util.cc:274] Using random seed: 1254170236
I20250110 20:53:51.694794 416 test_util.cc:274] Using random seed: 1254575992
I20250110 20:53:52.159955 8259 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0095250bbe784e2da1ba34c4537900af)
I20250110 20:53:52.160141 8259 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:52.161563 8259 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337), 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
I20250110 20:53:52.162936 8262 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0095250bbe784e2da1ba34c4537900af)
I20250110 20:53:52.163204 8262 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:52.164891 8262 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561), 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
I20250110 20:53:52.168695 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "deae654cdde442b39e56ee12665b00b5" is_pre_election: true
I20250110 20:53:52.168915 7913 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18c6b6ae45d6423c9b540c94c2266b3f in term 1.
I20250110 20:53:52.169363 8110 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6b6ae45d6423c9b540c94c2266b3f, deae654cdde442b39e56ee12665b00b5; no voters:
I20250110 20:53:52.169714 8259 raft_consensus.cc:2798] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:53:52.169824 8259 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0095250bbe784e2da1ba34c4537900af)
I20250110 20:53:52.169927 8259 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:52.171437 8174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "deae654cdde442b39e56ee12665b00b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" is_pre_election: true
I20250110 20:53:52.172195 8259 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:52.172588 8174 raft_consensus.cc:2388] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate deae654cdde442b39e56ee12665b00b5 in current term 2: Already voted for candidate 18c6b6ae45d6423c9b540c94c2266b3f in this term.
I20250110 20:53:52.172783 8259 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 2 election: Requested vote from peers deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337), 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
I20250110 20:53:52.173218 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "deae654cdde442b39e56ee12665b00b5"
I20250110 20:53:52.173388 7913 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:52.175215 7913 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c6b6ae45d6423c9b540c94c2266b3f in term 2.
I20250110 20:53:52.175526 8110 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 18c6b6ae45d6423c9b540c94c2266b3f, deae654cdde442b39e56ee12665b00b5; no voters:
I20250110 20:53:52.175760 8259 raft_consensus.cc:2798] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:53:52.176095 8259 raft_consensus.cc:695] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 LEADER]: Becoming Leader. State: Replica: 18c6b6ae45d6423c9b540c94c2266b3f, State: Running, Role: LEADER
I20250110 20:53:52.176502 8259 consensus_queue.cc:237] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:52.179450 7772 catalog_manager.cc:5526] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f reported cstate change: term changed from 1 to 2, leader changed from 0095250bbe784e2da1ba34c4537900af (127.0.104.2) to 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3). New cstate: current_term: 2 leader_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } } }
I20250110 20:53:52.200484 416 test_util.cc:274] Using random seed: 1255081719
I20250110 20:53:52.201951 416 test_util.cc:274] Using random seed: 1255083183
I20250110 20:53:52.214488 416 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250110 20:53:52.215613 416 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20250110 20:53:52.235709 7913 raft_consensus.cc:1270] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Refusing update from remote peer 18c6b6ae45d6423c9b540c94c2266b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250110 20:53:52.236485 8259 consensus_queue.cc:1035] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:53:52.293155 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling FlushMRSOp(b287ad5536de4f10929a2dbe394a3bc3): 1577950 bytes log retention, and flush 2998537 bytes memory
I20250110 20:53:52.364305 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: FlushMRSOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.070s user 0.052s sys 0.000s Metrics: {"bytes_written":135803,"cfile_init":1,"compiler_manager_pool.queue_time_us":454,"dirs.queue_time_us":841,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":3941,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":965,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":14,"spinlock_wait_cycles":577920,"thread_start_us":3622,"threads_started":2}
I20250110 20:53:52.366086 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling LogGCOp(b287ad5536de4f10929a2dbe394a3bc3): free 1577950 bytes of WAL
I20250110 20:53:52.367022 7850 log_reader.cc:385] T b287ad5536de4f10929a2dbe394a3bc3: removed 2 log segments from log reader
I20250110 20:53:52.367228 7850 log.cc:1078] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Deleting log segment in path: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/b287ad5536de4f10929a2dbe394a3bc3/wal-000000001 (ops 1-8)
I20250110 20:53:52.367452 7850 log.cc:1078] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Deleting log segment in path: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/b287ad5536de4f10929a2dbe394a3bc3/wal-000000002 (ops 9-14)
I20250110 20:53:52.368252 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: LogGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {}
I20250110 20:53:52.369329 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling FlushMRSOp(b287ad5536de4f10929a2dbe394a3bc3): 2498244 bytes log retention, and flush 2998537 bytes memory
I20250110 20:53:52.385977 8220 maintenance_manager.cc:418] P 18c6b6ae45d6423c9b540c94c2266b3f: Scheduling FlushMRSOp(b287ad5536de4f10929a2dbe394a3bc3): 4602274 bytes log retention, and flush 5079305 bytes memory
I20250110 20:53:52.392985 416 raft_consensus-itest-base.cc:274] Waiting for log GC on 18c6b6ae45d6423c9b540c94c2266b3f
I20250110 20:53:52.414548 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: FlushMRSOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.045s user 0.039s sys 0.000s Metrics: {"bytes_written":135869,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1095,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":860,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":17}
I20250110 20:53:52.415854 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling LogGCOp(b287ad5536de4f10929a2dbe394a3bc3): free 920370 bytes of WAL
I20250110 20:53:52.416198 7850 log_reader.cc:385] T b287ad5536de4f10929a2dbe394a3bc3: removed 1 log segments from log reader
I20250110 20:53:52.416313 7850 log.cc:1078] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5: Deleting log segment in path: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/b287ad5536de4f10929a2dbe394a3bc3/wal-000000003 (ops 15-21)
I20250110 20:53:52.416769 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: LogGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250110 20:53:52.417153 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3): 633 bytes on disk
I20250110 20:53:52.418143 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":2,"lbm_read_time_us":165,"lbm_reads_lt_1ms":8}
I20250110 20:53:52.418897 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling CompactRowSetsOp(b287ad5536de4f10929a2dbe394a3bc3): perf score=0.972787
I20250110 20:53:52.520454 8111 maintenance_manager.cc:642] P 18c6b6ae45d6423c9b540c94c2266b3f: FlushMRSOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.134s user 0.102s sys 0.021s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":10428,"dirs.queue_time_us":6656,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":841,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":34,"thread_start_us":17019,"threads_started":2}
I20250110 20:53:52.522130 8220 maintenance_manager.cc:418] P 18c6b6ae45d6423c9b540c94c2266b3f: Scheduling LogGCOp(b287ad5536de4f10929a2dbe394a3bc3): free 2761400 bytes of WAL
I20250110 20:53:52.522596 8111 log_reader.cc:385] T b287ad5536de4f10929a2dbe394a3bc3: removed 3 log segments from log reader
I20250110 20:53:52.522782 8111 log.cc:1078] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Deleting log segment in path: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/wals/b287ad5536de4f10929a2dbe394a3bc3/wal-000000001 (ops 1-9)
I20250110 20:53:52.522962 8111 log.cc:1078] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Deleting log segment in path: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/wals/b287ad5536de4f10929a2dbe394a3bc3/wal-000000002 (ops 10-16)
I20250110 20:53:52.523108 8111 log.cc:1078] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Deleting log segment in path: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/wals/b287ad5536de4f10929a2dbe394a3bc3/wal-000000003 (ops 17-23)
I20250110 20:53:52.524155 8111 maintenance_manager.cc:642] P 18c6b6ae45d6423c9b540c94c2266b3f: LogGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.002s user 0.000s sys 0.001s Metrics: {}
I20250110 20:53:52.524560 8220 maintenance_manager.cc:418] P 18c6b6ae45d6423c9b540c94c2266b3f: Scheduling UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3): 527 bytes on disk
I20250110 20:53:52.525385 8111 maintenance_manager.cc:642] P 18c6b6ae45d6423c9b540c94c2266b3f: UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20250110 20:53:52.526453 416 raft_consensus-itest-base.cc:279] Log GC complete on 18c6b6ae45d6423c9b540c94c2266b3f
I20250110 20:53:52.535220 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: CompactRowSetsOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.116s user 0.103s sys 0.012s Metrics: {"bytes_written":135937,"cfile_cache_hit":10,"cfile_cache_hit_bytes":262392,"cfile_cache_miss":26,"cfile_cache_miss_bytes":264206,"cfile_init":5,"delta_iterators_relevant":4,"dirs.queue_time_us":1300,"dirs.run_cpu_time_us":964,"dirs.run_wall_time_us":4091,"drs_written":1,"lbm_read_time_us":1211,"lbm_reads_lt_1ms":46,"lbm_write_time_us":943,"lbm_writes_lt_1ms":23,"num_input_rowsets":2,"peak_mem_usage":224,"rows_written":31,"thread_start_us":18967,"threads_started":2}
I20250110 20:53:52.536170 7959 maintenance_manager.cc:418] P deae654cdde442b39e56ee12665b00b5: Scheduling UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3): 492 bytes on disk
I20250110 20:53:52.536813 7850 maintenance_manager.cc:642] P deae654cdde442b39e56ee12665b00b5: UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4}
W20250110 20:53:53.801762 8110 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.2:43797 for RPC RequestConsensusVote timed out after 1.638s (ON_OUTBOUND_QUEUE)
W20250110 20:53:54.104094 7849 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.0.104.2:43797 for RPC RequestConsensusVote timed out after 1.937s (ON_OUTBOUND_QUEUE)
I20250110 20:53:54.527964 416 raft_consensus-itest-base.cc:295] Servers converged with original term 2
I20250110 20:53:54.528072 416 raft_consensus-itest-base.cc:300] Resuming 0095250bbe784e2da1ba34c4537900af
I20250110 20:53:54.531097 8173 raft_consensus.cc:1235] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 LEADER]: Rejecting Update request from peer 0095250bbe784e2da1ba34c4537900af for earlier term 1. Current term is 2. Ops: []
I20250110 20:53:54.531273 7913 raft_consensus.cc:1235] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Rejecting Update request from peer 0095250bbe784e2da1ba34c4537900af for earlier term 1. Current term is 2. Ops: []
I20250110 20:53:54.532037 8245 consensus_queue.cc:1046] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 38, Time since last communication: 0.003s
I20250110 20:53:54.535794 8245 consensus_queue.cc:1225] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [LEADER]: Peer 18c6b6ae45d6423c9b540c94c2266b3f log is divergent from this leader: its last log entry 2.38 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 38
I20250110 20:53:54.535950 8248 consensus_queue.cc:1046] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 38, Time since last communication: 0.000s
I20250110 20:53:54.540897 8305 raft_consensus.cc:3049] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:53:54.541009 8305 raft_consensus.cc:738] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0095250bbe784e2da1ba34c4537900af, State: Running, Role: LEADER
I20250110 20:53:54.541568 8305 consensus_queue.cc:260] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } }
I20250110 20:53:54.542174 8305 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:54.547024 8043 raft_consensus.cc:1270] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Refusing update from remote peer 18c6b6ae45d6423c9b540c94c2266b3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20250110 20:53:54.547040 8039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0095250bbe784e2da1ba34c4537900af" is_pre_election: true
I20250110 20:53:54.547394 8039 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18c6b6ae45d6423c9b540c94c2266b3f in term 2.
I20250110 20:53:54.547768 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0095250bbe784e2da1ba34c4537900af"
I20250110 20:53:54.548749 8039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "deae654cdde442b39e56ee12665b00b5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0095250bbe784e2da1ba34c4537900af" is_pre_election: true
I20250110 20:53:54.549242 7849 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: deae654cdde442b39e56ee12665b00b5; no voters: 0095250bbe784e2da1ba34c4537900af, 18c6b6ae45d6423c9b540c94c2266b3f
I20250110 20:53:54.550052 8307 raft_consensus.cc:2743] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:53:54.550330 8040 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c6b6ae45d6423c9b540c94c2266b3f in term 2.
I20250110 20:53:54.550567 8303 consensus_queue.cc:1035] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250110 20:53:54.551241 8303 consensus_queue.cc:740] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: The logs necessary to catch up peer 0095250bbe784e2da1ba34c4537900af have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..31: Segment 1 which contained index 2 has been GCed)
I20250110 20:53:54.551925 8303 raft_consensus.cc:1007] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Attempting to remove follower 0095250bbe784e2da1ba34c4537900af from the Raft config. Reason: The logs necessary to catch up peer 0095250bbe784e2da1ba34c4537900af have been garbage collected. The replica will never be able to catch up
I20250110 20:53:54.552798 8303 consensus_queue.cc:237] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 38, Committed index: 38, Last appended: 2.38, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } }
I20250110 20:53:54.554268 7913 raft_consensus.cc:1270] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Refusing update from remote peer 18c6b6ae45d6423c9b540c94c2266b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 38. Preceding OpId from leader: term: 2 index: 39. (index mismatch)
I20250110 20:53:54.554715 8308 consensus_queue.cc:1035] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 39, Last known committed idx: 38, Time since last communication: 0.000s
I20250110 20:53:54.556928 8303 raft_consensus.cc:2949] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 LEADER]: Committing config change with OpId 2.39: config changed from index -1 to 39, VOTER 0095250bbe784e2da1ba34c4537900af (127.0.104.2) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } }
I20250110 20:53:54.557402 7913 raft_consensus.cc:2949] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Committing config change with OpId 2.39: config changed from index -1 to 39, VOTER 0095250bbe784e2da1ba34c4537900af (127.0.104.2) evicted. New config: { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } }
I20250110 20:53:54.560755 7769 catalog_manager.cc:5526] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f reported cstate change: config changed from index -1 to 39, VOTER 0095250bbe784e2da1ba34c4537900af (127.0.104.2) evicted. New cstate: current_term: 2 leader_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" committed_config { opid_index: 39 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } }
I20250110 20:53:54.565579 8024 tablet_service.cc:1514] Processing DeleteTablet for tablet b287ad5536de4f10929a2dbe394a3bc3 with delete_type TABLET_DATA_TOMBSTONED (TS 0095250bbe784e2da1ba34c4537900af not found in new config with opid_index 39) from {username='slave'} at 127.0.0.1:50622
I20250110 20:53:54.566360 8315 tablet_replica.cc:331] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: stopping tablet replica
I20250110 20:53:54.566776 8315 raft_consensus.cc:2238] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:53:54.567085 8315 raft_consensus.cc:2267] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:53:54.568337 8315 ts_tablet_manager.cc:1905] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:53:54.574697 8315 ts_tablet_manager.cc:1918] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:53:54.574846 8315 log.cc:1198] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/b287ad5536de4f10929a2dbe394a3bc3
I20250110 20:53:54.575650 7759 catalog_manager.cc:4872] TS 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797): tablet b287ad5536de4f10929a2dbe394a3bc3 (table TestTable [id=9bc3bedc4316457eb9bf9e6bc6f26aab]) successfully deleted
I20250110 20:53:54.579275 8084 generic_service.cc:175] {username='slave'} at 127.0.0.1:50626 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20250110 20:53:54.580854 8173 consensus_queue.cc:237] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 39, Committed index: 39, Last appended: 2.39, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:54.582446 7913 raft_consensus.cc:1270] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Refusing update from remote peer 18c6b6ae45d6423c9b540c94c2266b3f: Log matching property violated. Preceding OpId in replica: term: 2 index: 39. Preceding OpId from leader: term: 2 index: 40. (index mismatch)
W20250110 20:53:54.582883 8110 consensus_peers.cc:487] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f -> Peer 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797): Couldn't send request to peer 0095250bbe784e2da1ba34c4537900af. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:53:54.583160 8308 consensus_queue.cc:1035] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 40, Last known committed idx: 39, Time since last communication: 0.000s
I20250110 20:53:54.584942 8303 raft_consensus.cc:2949] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 LEADER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 0095250bbe784e2da1ba34c4537900af (127.0.104.2) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } } }
I20250110 20:53:54.585512 7913 raft_consensus.cc:2949] T b287ad5536de4f10929a2dbe394a3bc3 P deae654cdde442b39e56ee12665b00b5 [term 2 FOLLOWER]: Committing config change with OpId 2.40: config changed from index 39 to 40, VOTER 0095250bbe784e2da1ba34c4537900af (127.0.104.2) added. New config: { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } } }
I20250110 20:53:54.589409 7769 catalog_manager.cc:5526] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f reported cstate change: config changed from index 39 to 40, VOTER 0095250bbe784e2da1ba34c4537900af (127.0.104.2) added. New cstate: current_term: 2 leader_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } } }
I20250110 20:53:55.086769 8316 ts_tablet_manager.cc:927] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Initiating tablet copy from peer 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561)
I20250110 20:53:55.087577 8316 tablet_copy_client.cc:323] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.3:42561
I20250110 20:53:55.088587 8194 tablet_copy_service.cc:140] P 18c6b6ae45d6423c9b540c94c2266b3f: Received BeginTabletCopySession request for tablet b287ad5536de4f10929a2dbe394a3bc3 from peer 0095250bbe784e2da1ba34c4537900af ({username='slave'} at 127.0.104.2:37165)
I20250110 20:53:55.088811 8194 tablet_copy_service.cc:161] P 18c6b6ae45d6423c9b540c94c2266b3f: Beginning new tablet copy session on tablet b287ad5536de4f10929a2dbe394a3bc3 from peer 0095250bbe784e2da1ba34c4537900af at {username='slave'} at 127.0.104.2:37165: session id = 0095250bbe784e2da1ba34c4537900af-b287ad5536de4f10929a2dbe394a3bc3
I20250110 20:53:55.091790 8194 tablet_copy_source_session.cc:215] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Tablet Copy: opened 5 blocks and 3 log segments
I20250110 20:53:55.093735 8316 ts_tablet_manager.cc:1905] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 20:53:55.098024 8316 ts_tablet_manager.cc:1918] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20250110 20:53:55.098315 8316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b287ad5536de4f10929a2dbe394a3bc3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:53:55.100579 8316 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250110 20:53:55.100666 8316 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20250110 20:53:55.100732 8316 fault_injection.cc:59] injecting fault for kudu (pid 7962): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20250110 20:53:55.139858 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 7831
I20250110 20:53:55.140941 7953 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:55.251515 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 7831
I20250110 20:53:55.269989 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8092
I20250110 20:53:55.270963 8214 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:55.382616 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8092
I20250110 20:53:55.400718 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 7741
I20250110 20:53:55.401723 7802 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:53:55.502532 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 7741
I20250110 20:53:55.520929 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44373
--webserver_interface=127.0.104.62
--webserver_port=36231
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44373
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20250110 20:53:55.825372 8321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:55.825654 8321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:55.825778 8321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:55.833057 8321 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:55.833197 8321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:53:55.833261 8321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:55.833338 8321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:53:55.833400 8321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:53:55.842217 8321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44373
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=36231
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:55.843674 8321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:55.845218 8321 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:55.854566 8327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:55.854589 8326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:55.854780 8329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:55.855249 8321 server_base.cc:1034] running on GCE node
I20250110 20:53:55.855886 8321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:55.857079 8321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:55.858263 8321 hybrid_clock.cc:648] HybridClock initialized: now 1736542435858223 us; error 36 us; skew 500 ppm
I20250110 20:53:55.858558 8321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:55.865609 8321 webserver.cc:458] Webserver started at http://127.0.104.62:36231/ using document root <none> and password file <none>
I20250110 20:53:55.866271 8321 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:55.866389 8321 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:55.871124 8321 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20250110 20:53:55.873570 8335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:55.874787 8321 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250110 20:53:55.874931 8321 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1a4f2670dfce41239d1db9bfeef9d76f"
format_stamp: "Formatted at 2025-01-10 20:53:48 on dist-test-slave-1h8z"
I20250110 20:53:55.875667 8321 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:55.894335 8321 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:55.895104 8321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:55.895365 8321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:55.911018 8321 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44373
I20250110 20:53:55.911047 8386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44373 every 8 connection(s)
I20250110 20:53:55.912367 8321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:53:55.918130 8387 sys_catalog.cc:263] Verifying existing consensus state
I20250110 20:53:55.920029 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8321
I20250110 20:53:55.921073 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:42561
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:44373
--webserver_port=37277
--webserver_interface=127.0.104.3
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250110 20:53:55.921474 8387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Bootstrap starting.
I20250110 20:53:55.938143 8387 log.cc:826] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:55.950138 8387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=5 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 20:53:55.950599 8387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Bootstrap complete.
I20250110 20:53:55.957892 8387 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:55.958573 8387 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a4f2670dfce41239d1db9bfeef9d76f, State: Initialized, Role: FOLLOWER
I20250110 20:53:55.959272 8387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:55.959475 8387 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:53:55.959578 8387 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:53:55.959726 8387 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:53:55.963156 8387 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:55.963616 8387 leader_election.cc:304] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a4f2670dfce41239d1db9bfeef9d76f; no voters:
I20250110 20:53:55.964147 8387 leader_election.cc:290] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250110 20:53:55.964303 8391 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:53:55.966213 8387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:53:55.971207 8391 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [term 2 LEADER]: Becoming Leader. State: Replica: 1a4f2670dfce41239d1db9bfeef9d76f, State: Running, Role: LEADER
I20250110 20:53:55.971758 8391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } }
I20250110 20:53:55.974305 8392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } } }
I20250110 20:53:55.974661 8392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:55.975152 8393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a4f2670dfce41239d1db9bfeef9d76f. Latest consensus state: current_term: 2 leader_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a4f2670dfce41239d1db9bfeef9d76f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44373 } } }
I20250110 20:53:55.975383 8393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f [sys.catalog]: This master's current role is: LEADER
I20250110 20:53:55.977180 8402 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:53:55.982424 8402 catalog_manager.cc:670] Loaded metadata for table TestTable [id=9bc3bedc4316457eb9bf9e6bc6f26aab]
I20250110 20:53:55.984653 8402 tablet_loader.cc:96] loaded metadata for tablet b287ad5536de4f10929a2dbe394a3bc3 (table TestTable [id=9bc3bedc4316457eb9bf9e6bc6f26aab])
I20250110 20:53:55.985078 8402 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:53:55.986439 8402 catalog_manager.cc:1260] Loaded cluster ID: dccc8f2f36a84c04bdec375a22cd8ad6
I20250110 20:53:55.986593 8402 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:53:55.990432 8402 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:53:55.991887 8402 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 1a4f2670dfce41239d1db9bfeef9d76f: Loaded TSK: 0
I20250110 20:53:55.993280 8402 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250110 20:53:56.246034 8389 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250110 20:53:56.246397 8389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:56.246481 8389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:56.246591 8389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:56.257153 8389 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250110 20:53:56.257247 8389 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250110 20:53:56.257361 8389 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:56.257477 8389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:56.257660 8389 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250110 20:53:56.257737 8389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:53:56.271543 8389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:42561
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=37277
--tserver_master_addrs=127.0.104.62:44373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:56.273475 8389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:56.275089 8389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:56.286672 8413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:56.287495 8414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:56.289108 8416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:56.289076 8389 server_base.cc:1034] running on GCE node
I20250110 20:53:56.289716 8389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:56.290580 8389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:56.291777 8389 hybrid_clock.cc:648] HybridClock initialized: now 1736542436291733 us; error 39 us; skew 500 ppm
I20250110 20:53:56.292037 8389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:56.294204 8389 webserver.cc:458] Webserver started at http://127.0.104.3:37277/ using document root <none> and password file <none>
I20250110 20:53:56.294911 8389 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:56.295010 8389 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:56.299877 8389 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20250110 20:53:56.308202 8425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:56.309319 8389 fs_manager.cc:730] Time spent opening block manager: real 0.008s user 0.002s sys 0.001s
I20250110 20:53:56.309468 8389 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "18c6b6ae45d6423c9b540c94c2266b3f"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:56.310223 8389 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 5
Total live bytes: 136528
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:56.344103 8389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:56.344873 8389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:56.345117 8389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:56.345988 8389 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:56.347756 8432 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250110 20:53:56.352842 8389 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250110 20:53:56.352916 8389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20250110 20:53:56.353027 8389 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250110 20:53:56.355634 8389 ts_tablet_manager.cc:610] Registered 1 tablets
I20250110 20:53:56.355716 8389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.002s sys 0.000s
I20250110 20:53:56.356036 8432 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Bootstrap starting.
I20250110 20:53:56.373454 8432 log.cc:826] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:56.388429 8432 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=6} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 1 replicates
I20250110 20:53:56.395195 8389 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42561
I20250110 20:53:56.395262 8539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42561 every 8 connection(s)
I20250110 20:53:56.396261 8432 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=11 ignored=11} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 3 replicates
I20250110 20:53:56.396432 8389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:53:56.404152 8432 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Bootstrap replayed 3/3 log segments. Stats: ops{read=17 overwritten=0 applied=17 ignored=13} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates
I20250110 20:53:56.404742 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8389
I20250110 20:53:56.404927 8432 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Bootstrap complete.
I20250110 20:53:56.405787 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:43797
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:44373
--webserver_port=41429
--webserver_interface=127.0.104.2
--builtin_ntp_servers=127.0.104.20:44291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250110 20:53:56.406031 8432 ts_tablet_manager.cc:1397] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Time spent bootstrapping tablet: real 0.050s user 0.031s sys 0.016s
I20250110 20:53:56.414302 8432 raft_consensus.cc:357] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:56.416432 8432 raft_consensus.cc:738] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18c6b6ae45d6423c9b540c94c2266b3f, State: Initialized, Role: FOLLOWER
I20250110 20:53:56.417223 8432 consensus_queue.cc:260] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:56.427181 8432 ts_tablet_manager.cc:1428] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Time spent starting tablet: real 0.021s user 0.012s sys 0.009s
I20250110 20:53:56.430390 8540 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44373
I20250110 20:53:56.430634 8540 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:56.431269 8540 heartbeater.cc:510] Master 127.0.104.62:44373 requested a full tablet report, sending...
I20250110 20:53:56.434216 8352 ts_manager.cc:194] Registered new tserver with Master: 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561)
I20250110 20:53:56.437747 8352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:55583
I20250110 20:53:56.440975 8540 heartbeater.cc:502] Master 127.0.104.62:44373 was elected leader, sending a full tablet report...
I20250110 20:53:56.517079 8541 maintenance_manager.cc:418] P 18c6b6ae45d6423c9b540c94c2266b3f: Scheduling UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3): 527 bytes on disk
I20250110 20:53:56.518596 8430 maintenance_manager.cc:642] P 18c6b6ae45d6423c9b540c94c2266b3f: UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
W20250110 20:53:56.723378 8544 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250110 20:53:56.723680 8544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:53:56.723743 8544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:53:56.723819 8544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:53:56.731204 8544 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250110 20:53:56.731279 8544 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250110 20:53:56.731355 8544 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250110 20:53:56.731431 8544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:53:56.731557 8544 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250110 20:53:56.731611 8544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:53:56.740475 8544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:43797
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=41429
--tserver_master_addrs=127.0.104.62:44373
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:53:56.741837 8544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:53:56.743372 8544 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:53:56.753485 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:56.753522 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:53:56.753548 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:53:56.754292 8544 server_base.cc:1034] running on GCE node
I20250110 20:53:56.754851 8544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:53:56.755599 8544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:53:56.756772 8544 hybrid_clock.cc:648] HybridClock initialized: now 1736542436756733 us; error 33 us; skew 500 ppm
I20250110 20:53:56.757026 8544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:53:56.759011 8544 webserver.cc:458] Webserver started at http://127.0.104.2:41429/ using document root <none> and password file <none>
I20250110 20:53:56.759670 8544 fs_manager.cc:362] Metadata directory not provided
I20250110 20:53:56.759770 8544 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:53:56.764286 8544 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20250110 20:53:56.766566 8559 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:56.767750 8544 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250110 20:53:56.767889 8544 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0095250bbe784e2da1ba34c4537900af"
format_stamp: "Formatted at 2025-01-10 20:53:49 on dist-test-slave-1h8z"
I20250110 20:53:56.768584 8544 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:53:56.787425 8544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:53:56.788087 8544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:53:56.788331 8544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:53:56.789108 8544 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:53:56.790791 8566 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20250110 20:53:56.795179 8566 ts_tablet_manager.cc:1852] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20250110 20:53:56.797670 8566 ts_tablet_manager.cc:1905] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:53:56.803900 8566 ts_tablet_manager.cc:1918] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:53:56.804898 8544 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20250110 20:53:56.804991 8544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20250110 20:53:56.805150 8544 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:53:56.805202 8544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:53:56.833165 8544 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43797
I20250110 20:53:56.833206 8672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43797 every 8 connection(s)
I20250110 20:53:56.834347 8544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:53:56.844606 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8544
I20250110 20:53:56.844880 416 test_util.cc:274] Using random seed: 1259726137
I20250110 20:53:56.845865 8673 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44373
I20250110 20:53:56.846107 8673 heartbeater.cc:463] Registering TS with master...
I20250110 20:53:56.846793 8673 heartbeater.cc:510] Master 127.0.104.62:44373 requested a full tablet report, sending...
I20250110 20:53:56.848452 8352 ts_manager.cc:194] Registered new tserver with Master: 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
I20250110 20:53:56.849442 8352 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:47183
I20250110 20:53:56.858922 416 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
W20250110 20:53:56.899034 8454 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.898989 8452 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.899169 8453 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.899797 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.902427 8679 meta_cache.cc:302] tablet b287ad5536de4f10929a2dbe394a3bc3: replica deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337) has failed: Network error: Client connection negotiation failed: client connection to 127.0.104.1:44337: connect: Connection refused (error 111)
W20250110 20:53:56.917713 8453 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.918131 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.924062 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.932158 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.945412 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.957194 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.963074 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.976097 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:56.985975 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.000154 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.014955 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.034114 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.053014 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.069238 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.087468 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.108000 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.115159 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.140538 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.175585 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.199519 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.209656 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.230841 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.252951 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.280283 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.301450 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.319742 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.343972 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.368352 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.393754 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.410094 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.439390 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.464604 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.492923 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.510237 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.543707 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.571382 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.608189 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.636576 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.658080 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.695991 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.724900 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.751610 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.783128 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.817580 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.847167 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
I20250110 20:53:57.851562 8673 heartbeater.cc:502] Master 127.0.104.62:44373 was elected leader, sending a full tablet report...
I20250110 20:53:57.878511 8688 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:53:57.878733 8688 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:57.880231 8688 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337), 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
W20250110 20:53:57.881785 8429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.1:44337: connect: Connection refused (error 111)
W20250110 20:53:57.882608 8451 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33432: Illegal state: replica 18c6b6ae45d6423c9b540c94c2266b3f is not leader of this config: current role FOLLOWER
W20250110 20:53:57.883713 8429 leader_election.cc:336] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337): Network error: Client connection negotiation failed: client connection to 127.0.104.1:44337: connect: Connection refused (error 111)
I20250110 20:53:57.887168 8628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: false dest_uuid: "0095250bbe784e2da1ba34c4537900af" is_pre_election: true
I20250110 20:53:57.889267 8628 raft_consensus.cc:1798] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20250110 20:53:57.889873 8628 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 18c6b6ae45d6423c9b540c94c2266b3f in term 2.
I20250110 20:53:57.890306 8429 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0095250bbe784e2da1ba34c4537900af, 18c6b6ae45d6423c9b540c94c2266b3f; no voters: deae654cdde442b39e56ee12665b00b5
I20250110 20:53:57.890630 8688 raft_consensus.cc:2798] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:53:57.890799 8688 raft_consensus.cc:491] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:53:57.890902 8688 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:53:57.893903 8688 raft_consensus.cc:513] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:57.894398 8688 leader_election.cc:290] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 3 election: Requested vote from peers deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337), 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797)
I20250110 20:53:57.894958 8628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b287ad5536de4f10929a2dbe394a3bc3" candidate_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" candidate_term: 3 candidate_status { last_received { term: 2 index: 40 } } ignore_live_leader: false dest_uuid: "0095250bbe784e2da1ba34c4537900af"
I20250110 20:53:57.895107 8628 raft_consensus.cc:1798] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20250110 20:53:57.895205 8628 raft_consensus.cc:3054] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 2 FOLLOWER]: Advancing to term 3
W20250110 20:53:57.895404 8429 leader_election.cc:336] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337): Network error: Client connection negotiation failed: client connection to 127.0.104.1:44337: connect: Connection refused (error 111)
I20250110 20:53:57.897176 8628 raft_consensus.cc:2463] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 18c6b6ae45d6423c9b540c94c2266b3f in term 3.
I20250110 20:53:57.897500 8429 leader_election.cc:304] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0095250bbe784e2da1ba34c4537900af, 18c6b6ae45d6423c9b540c94c2266b3f; no voters: deae654cdde442b39e56ee12665b00b5
I20250110 20:53:57.897753 8688 raft_consensus.cc:2798] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 3 FOLLOWER]: Leader election won for term 3
I20250110 20:53:57.898063 8688 raft_consensus.cc:695] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [term 3 LEADER]: Becoming Leader. State: Replica: 18c6b6ae45d6423c9b540c94c2266b3f, State: Running, Role: LEADER
I20250110 20:53:57.898504 8688 consensus_queue.cc:237] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 40, Committed index: 40, Last appended: 2.40, Last appended by leader: 40, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:57.901890 8343 catalog_manager.cc:5526] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" committed_config { opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } } }
W20250110 20:53:57.921535 8429 consensus_peers.cc:487] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f -> Peer 0095250bbe784e2da1ba34c4537900af (127.0.104.2:43797): Couldn't send request to peer 0095250bbe784e2da1ba34c4537900af. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:53:57.921700 8429 consensus_peers.cc:487] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f -> Peer deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337): Couldn't send request to peer deae654cdde442b39e56ee12665b00b5. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.1:44337: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:53:57.928316 8679 meta_cache.cc:302] tablet b287ad5536de4f10929a2dbe394a3bc3: replica deae654cdde442b39e56ee12665b00b5 (127.0.104.1:44337) has failed: Network error: Client connection negotiation failed: client connection to 127.0.104.1:44337: connect: Connection refused (error 111)
I20250110 20:53:58.410974 8700 tablet_replica.cc:331] stopping tablet replica
I20250110 20:53:58.411201 8700 raft_consensus.cc:2238] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 3 FOLLOWER]: Raft consensus shutting down.
I20250110 20:53:58.411381 8700 raft_consensus.cc:2267] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 3 FOLLOWER]: Raft consensus is shut down!
I20250110 20:53:58.411643 8700 ts_tablet_manager.cc:927] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Initiating tablet copy from peer 18c6b6ae45d6423c9b540c94c2266b3f (127.0.104.3:42561)
I20250110 20:53:58.412410 8700 tablet_copy_client.cc:323] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.3:42561
I20250110 20:53:58.418252 8515 tablet_copy_service.cc:140] P 18c6b6ae45d6423c9b540c94c2266b3f: Received BeginTabletCopySession request for tablet b287ad5536de4f10929a2dbe394a3bc3 from peer 0095250bbe784e2da1ba34c4537900af ({username='slave'} at 127.0.104.2:60535)
I20250110 20:53:58.418426 8515 tablet_copy_service.cc:161] P 18c6b6ae45d6423c9b540c94c2266b3f: Beginning new tablet copy session on tablet b287ad5536de4f10929a2dbe394a3bc3 from peer 0095250bbe784e2da1ba34c4537900af at {username='slave'} at 127.0.104.2:60535: session id = 0095250bbe784e2da1ba34c4537900af-b287ad5536de4f10929a2dbe394a3bc3
I20250110 20:53:58.421039 8515 tablet_copy_source_session.cc:215] T b287ad5536de4f10929a2dbe394a3bc3 P 18c6b6ae45d6423c9b540c94c2266b3f: Tablet Copy: opened 5 blocks and 3 log segments
I20250110 20:53:58.423230 8700 ts_tablet_manager.cc:1905] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250110 20:53:58.427413 8700 ts_tablet_manager.cc:1918] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20250110 20:53:58.427781 8700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b287ad5536de4f10929a2dbe394a3bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:53:58.430112 8700 tablet_copy_client.cc:806] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet copy: Starting download of 5 data blocks...
I20250110 20:53:58.438704 8700 tablet_copy_client.cc:670] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet copy: Starting download of 3 WAL segments...
I20250110 20:53:58.448035 8700 tablet_copy_client.cc:538] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250110 20:53:58.451043 8700 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Bootstrap starting.
I20250110 20:53:58.460942 8700 log.cc:826] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Log is configured to *not* fsync() on all Append() calls
I20250110 20:53:58.471006 8700 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=6} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250110 20:53:58.477187 8700 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=11 ignored=11} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250110 20:53:58.483776 8700 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Bootstrap replayed 3/3 log segments. Stats: ops{read=22 overwritten=0 applied=17 ignored=13} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20250110 20:53:58.484239 8700 tablet_bootstrap.cc:492] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Bootstrap complete.
I20250110 20:53:58.484875 8700 ts_tablet_manager.cc:1397] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.000s
I20250110 20:53:58.487357 8700 raft_consensus.cc:357] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 3 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:58.489900 8700 raft_consensus.cc:738] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0095250bbe784e2da1ba34c4537900af, State: Initialized, Role: FOLLOWER
I20250110 20:53:58.490448 8700 consensus_queue.cc:260] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 40, Last appended: 3.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 40 OBSOLETE_local: false peers { permanent_uuid: "deae654cdde442b39e56ee12665b00b5" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 44337 } } peers { permanent_uuid: "18c6b6ae45d6423c9b540c94c2266b3f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42561 } } peers { permanent_uuid: "0095250bbe784e2da1ba34c4537900af" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43797 } attrs { } }
I20250110 20:53:58.491794 8700 ts_tablet_manager.cc:1428] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20250110 20:53:58.492758 8514 tablet_copy_service.cc:342] P 18c6b6ae45d6423c9b540c94c2266b3f: Request end of tablet copy session 0095250bbe784e2da1ba34c4537900af-b287ad5536de4f10929a2dbe394a3bc3 received from {username='slave'} at 127.0.104.2:60535
I20250110 20:53:58.492923 8514 tablet_copy_service.cc:434] P 18c6b6ae45d6423c9b540c94c2266b3f: ending tablet copy session 0095250bbe784e2da1ba34c4537900af-b287ad5536de4f10929a2dbe394a3bc3 on tablet b287ad5536de4f10929a2dbe394a3bc3 with peer 0095250bbe784e2da1ba34c4537900af
I20250110 20:53:58.538991 8674 maintenance_manager.cc:418] P 0095250bbe784e2da1ba34c4537900af: Scheduling UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3): 527 bytes on disk
I20250110 20:53:58.540131 8564 maintenance_manager.cc:642] P 0095250bbe784e2da1ba34c4537900af: UndoDeltaBlockGCOp(b287ad5536de4f10929a2dbe394a3bc3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20250110 20:53:59.001865 8628 raft_consensus.cc:1212] T b287ad5536de4f10929a2dbe394a3bc3 P 0095250bbe784e2da1ba34c4537900af [term 3 FOLLOWER]: Deduplicated request from leader. Original: 2.40->[3.41-3.45] Dedup: 3.45->[]
I20250110 20:53:59.007831 8713 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33432 (ReqId={client: 719aaef076014e899276ff15140c05b0, seq_no=0, attempt_no=41}) took 1090 ms. Trace:
I20250110 20:53:59.009110 8714 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33432 (ReqId={client: 719aaef076014e899276ff15140c05b0, seq_no=2, attempt_no=42}) took 1053 ms. Trace:
I20250110 20:53:59.008090 8713 rpcz_store.cc:276] 0110 20:53:57.917645 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:57.917705 (+ 60us) service_pool.cc:225] Handling call
0110 20:53:59.007678 (+1089973us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:57.918495 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b287ad5536de4f10929a2dbe394a3bc3
0110 20:53:57.918638 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:57.918642 (+ 4us) write_op.cc:435] Acquired schema lock
0110 20:53:57.918644 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:57.919089 (+ 445us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:57.919100 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:57.919104 (+ 4us) tablet.cc:682] Acquiring locks for 50 operations
0110 20:53:57.919351 (+ 247us) tablet.cc:698] Row locks acquired
0110 20:53:57.919353 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:57.919407 (+ 54us) write_op.cc:270] Start()
0110 20:53:57.919442 (+ 35us) write_op.cc:276] Timestamp: P: 1736542437919372 usec, L: 0
0110 20:53:57.919446 (+ 4us) op_driver.cc:348] REPLICATION: starting
0110 20:53:57.919625 (+ 179us) log.cc:844] Serialized 2455 byte log entry
0110 20:53:59.004333 (+1084708us) op_driver.cc:464] REPLICATION: finished
0110 20:53:59.004801 (+ 468us) write_op.cc:301] APPLY: starting
0110 20:53:59.004819 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:59.006058 (+ 1239us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:59.006085 (+ 27us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:59.006661 (+ 576us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:59.006867 (+ 206us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=48,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:59.006893 (+ 26us) write_op.cc:312] APPLY: finished
0110 20:53:59.007091 (+ 198us) log.cc:844] Serialized 319 byte log entry
0110 20:53:59.007420 (+ 329us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:59.007474 (+ 54us) write_op.cc:454] Released schema lock
0110 20:53:59.007488 (+ 14us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":374,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":6,"mutex_wait_us":37,"num_ops":50,"prepare.queue_time_us":374,"prepare.run_cpu_time_us":1294,"prepare.run_wall_time_us":1295,"raft.queue_time_us":97,"raft.run_cpu_time_us":848,"raft.run_wall_time_us":863,"replication_time_us":1084836,"spinlock_wait_cycles":329600,"thread_start_us":570,"threads_started":2}]]}
I20250110 20:53:59.009263 8714 rpcz_store.cc:276] 0110 20:53:57.955442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:53:57.955508 (+ 66us) service_pool.cc:225] Handling call
0110 20:53:59.009014 (+1053506us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:53:57.955971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet b287ad5536de4f10929a2dbe394a3bc3
0110 20:53:57.956064 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:53:57.956070 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:53:57.956072 (+ 2us) tablet.cc:659] Decoding operations
0110 20:53:57.956357 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:53:57.956385 (+ 28us) write_op.cc:641] Partition lock acquired for write op
0110 20:53:57.956389 (+ 4us) tablet.cc:682] Acquiring locks for 50 operations
0110 20:53:57.956597 (+ 208us) tablet.cc:698] Row locks acquired
0110 20:53:57.956599 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:53:57.956640 (+ 41us) write_op.cc:270] Start()
0110 20:53:57.956669 (+ 29us) write_op.cc:276] Timestamp: P: 1736542437956602 usec, L: 0
0110 20:53:57.956671 (+ 2us) op_driver.cc:348] REPLICATION: starting
0110 20:53:57.956838 (+ 167us) log.cc:844] Serialized 2455 byte log entry
0110 20:53:59.004733 (+1047895us) op_driver.cc:464] REPLICATION: finished
0110 20:53:59.007104 (+ 2371us) write_op.cc:301] APPLY: starting
0110 20:53:59.007119 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0110 20:53:59.007687 (+ 568us) tablet.cc:1367] finished BulkCheckPresence
0110 20:53:59.007757 (+ 70us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:53:59.008582 (+ 825us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:53:59.008645 (+ 63us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=48,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:53:59.008653 (+ 8us) write_op.cc:312] APPLY: finished
0110 20:53:59.008790 (+ 137us) log.cc:844] Serialized 319 byte log entry
0110 20:53:59.008858 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:53:59.008897 (+ 39us) write_op.cc:454] Released schema lock
0110 20:53:59.008906 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2300,"cfile_cache_hit":2,"cfile_cache_hit_bytes":4127,"mutex_wait_us":1,"num_ops":50,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":997,"prepare.run_wall_time_us":998,"raft.queue_time_us":146,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":14,"replication_time_us":1048035,"spinlock_wait_cycles":3328,"thread_start_us":2173,"threads_started":1}]]}
I20250110 20:53:59.897305 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8544
I20250110 20:53:59.903821 8668 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:00.005831 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8544
I20250110 20:54:00.021566 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8389
I20250110 20:54:00.027544 8535 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:00.128337 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8389
I20250110 20:54:00.144755 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8321
I20250110 20:54:00.150365 8382 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:00.247627 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8321
2025-01-10T20:54:00Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (12157 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20250110 20:54:00.284544 416 test_util.cc:274] Using random seed: 1263165804
I20250110 20:54:00.286164 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:54:00.286224 416 ts_itest-base.cc:113] --------------
I20250110 20:54:00.286290 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:54:00.286338 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:54:00.286403 416 ts_itest-base.cc:116] --------------
2025-01-10T20:54:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:54:00Z Disabled control of system clock
I20250110 20:54:00.314898 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44969
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44969 with env {}
W20250110 20:54:00.619843 8739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:00.620137 8739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:00.620268 8739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:00.627499 8739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:54:00.627588 8739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:00.627653 8739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:54:00.627720 8739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:54:00.636372 8739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44969
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:00.637735 8739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:00.639349 8739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:00.648514 8747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:00.648454 8745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:00.648548 8744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:00.649540 8739 server_base.cc:1034] running on GCE node
I20250110 20:54:00.650195 8739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:00.651472 8739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:00.652678 8739 hybrid_clock.cc:648] HybridClock initialized: now 1736542440652626 us; error 28 us; skew 500 ppm
I20250110 20:54:00.652957 8739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:00.654974 8739 webserver.cc:458] Webserver started at http://127.0.104.62:46381/ using document root <none> and password file <none>
I20250110 20:54:00.655615 8739 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:00.655723 8739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:00.655998 8739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:00.658432 8739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "63c240fea34b46ada5b167fbe8d37c46"
format_stamp: "Formatted at 2025-01-10 20:54:00 on dist-test-slave-1h8z"
I20250110 20:54:00.659138 8739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "63c240fea34b46ada5b167fbe8d37c46"
format_stamp: "Formatted at 2025-01-10 20:54:00 on dist-test-slave-1h8z"
I20250110 20:54:00.663655 8739 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250110 20:54:00.666491 8753 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:00.667591 8739 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250110 20:54:00.667737 8739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "63c240fea34b46ada5b167fbe8d37c46"
format_stamp: "Formatted at 2025-01-10 20:54:00 on dist-test-slave-1h8z"
I20250110 20:54:00.667882 8739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:00.679873 8739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:00.680562 8739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:00.680820 8739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:00.696476 8739 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44969
I20250110 20:54:00.696514 8804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44969 every 8 connection(s)
I20250110 20:54:00.697636 8739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:54:00.700948 8805 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:00.701401 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8739
I20250110 20:54:00.701555 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:54:00.712854 8805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46: Bootstrap starting.
I20250110 20:54:00.715692 8805 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:00.716573 8805 log.cc:826] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:00.718674 8805 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46: No bootstrap required, opened a new log
I20250110 20:54:00.725113 8805 raft_consensus.cc:357] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c240fea34b46ada5b167fbe8d37c46" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44969 } }
I20250110 20:54:00.725409 8805 raft_consensus.cc:383] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:54:00.725507 8805 raft_consensus.cc:738] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c240fea34b46ada5b167fbe8d37c46, State: Initialized, Role: FOLLOWER
I20250110 20:54:00.726006 8805 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c240fea34b46ada5b167fbe8d37c46" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44969 } }
I20250110 20:54:00.726179 8805 raft_consensus.cc:397] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:54:00.726290 8805 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:54:00.726423 8805 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:00.728302 8805 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c240fea34b46ada5b167fbe8d37c46" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44969 } }
I20250110 20:54:00.728716 8805 leader_election.cc:304] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63c240fea34b46ada5b167fbe8d37c46; no voters:
I20250110 20:54:00.729190 8805 leader_election.cc:290] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:00.729316 8810 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:00.729825 8810 raft_consensus.cc:695] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [term 1 LEADER]: Becoming Leader. State: Replica: 63c240fea34b46ada5b167fbe8d37c46, State: Running, Role: LEADER
I20250110 20:54:00.730265 8810 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c240fea34b46ada5b167fbe8d37c46" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44969 } }
I20250110 20:54:00.730866 8805 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:54:00.732859 8811 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63c240fea34b46ada5b167fbe8d37c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c240fea34b46ada5b167fbe8d37c46" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44969 } } }
I20250110 20:54:00.732887 8812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63c240fea34b46ada5b167fbe8d37c46. Latest consensus state: current_term: 1 leader_uuid: "63c240fea34b46ada5b167fbe8d37c46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c240fea34b46ada5b167fbe8d37c46" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44969 } } }
I20250110 20:54:00.733136 8811 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [sys.catalog]: This master's current role is: LEADER
I20250110 20:54:00.733183 8812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46 [sys.catalog]: This master's current role is: LEADER
I20250110 20:54:00.739392 8817 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:54:00.744163 8817 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:54:00.751420 8817 catalog_manager.cc:1348] Generated new cluster ID: b9e7784c4b4f48eabbcfe1d1e47e6bb7
I20250110 20:54:00.751523 8817 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:54:00.764978 8817 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:54:00.766038 8817 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:54:00.772272 8817 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 63c240fea34b46ada5b167fbe8d37c46: Generated new TSK 0
I20250110 20:54:00.772951 8817 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:54:00.778609 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44969
--builtin_ntp_servers=127.0.104.20:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20250110 20:54:01.108697 8829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:01.108961 8829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:01.109066 8829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:01.116528 8829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:01.116734 8829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:54:01.125401 8829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44969
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:01.126863 8829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:01.128448 8829 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:01.138849 8835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:01.138888 8837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:01.138911 8834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:01.139613 8829 server_base.cc:1034] running on GCE node
I20250110 20:54:01.140141 8829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:01.140957 8829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:01.142153 8829 hybrid_clock.cc:648] HybridClock initialized: now 1736542441142100 us; error 35 us; skew 500 ppm
I20250110 20:54:01.142419 8829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:01.149688 8829 webserver.cc:458] Webserver started at http://127.0.104.1:45545/ using document root <none> and password file <none>
I20250110 20:54:01.150391 8829 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:01.150499 8829 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:01.150795 8829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:01.153129 8829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.153810 8829 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.158275 8829 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20250110 20:54:01.161413 8843 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.162559 8829 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250110 20:54:01.162698 8829 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.162883 8829 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:01.196921 8829 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:01.197667 8829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:01.197919 8829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:01.198789 8829 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:01.200170 8829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:01.200250 8829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.200364 8829 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:01.200414 8829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.227245 8829 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:38959
I20250110 20:54:01.227303 8955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:38959 every 8 connection(s)
I20250110 20:54:01.228427 8829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:54:01.234913 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8829
I20250110 20:54:01.235116 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:54:01.238277 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44969
--builtin_ntp_servers=127.0.104.20:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20250110 20:54:01.240351 8956 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44969
I20250110 20:54:01.240655 8956 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:01.241508 8956 heartbeater.cc:510] Master 127.0.104.62:44969 requested a full tablet report, sending...
I20250110 20:54:01.243497 8770 ts_manager.cc:194] Registered new tserver with Master: da7d2e88cc74473ba3c7212ea6b5c9e8 (127.0.104.1:38959)
I20250110 20:54:01.244885 8770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:57129
W20250110 20:54:01.543033 8960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:01.543282 8960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:01.543406 8960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:01.550771 8960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:01.550976 8960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:54:01.559620 8960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44969
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:01.560982 8960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:01.562525 8960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:01.573000 8966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:01.572955 8965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:01.573050 8968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:01.573546 8960 server_base.cc:1034] running on GCE node
I20250110 20:54:01.574035 8960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:01.574800 8960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:01.575985 8960 hybrid_clock.cc:648] HybridClock initialized: now 1736542441575921 us; error 43 us; skew 500 ppm
I20250110 20:54:01.576244 8960 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:01.578105 8960 webserver.cc:458] Webserver started at http://127.0.104.2:35661/ using document root <none> and password file <none>
I20250110 20:54:01.578792 8960 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:01.578895 8960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:01.579164 8960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:01.581409 8960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "871a805f7e704954af58e3aa4ee77dc8"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.582059 8960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "871a805f7e704954af58e3aa4ee77dc8"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.586405 8960 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250110 20:54:01.589287 8974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.590389 8960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250110 20:54:01.590534 8960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "871a805f7e704954af58e3aa4ee77dc8"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.590672 8960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:01.606523 8960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:01.607285 8960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:01.607537 8960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:01.608398 8960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:01.609740 8960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:01.609807 8960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.609915 8960 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:01.609963 8960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.637126 8960 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42789
I20250110 20:54:01.637161 9086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42789 every 8 connection(s)
I20250110 20:54:01.638249 8960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:54:01.639640 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 8960
I20250110 20:54:01.639792 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:54:01.642869 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44969
--builtin_ntp_servers=127.0.104.20:34333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20250110 20:54:01.651103 9087 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44969
I20250110 20:54:01.651396 9087 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:01.652223 9087 heartbeater.cc:510] Master 127.0.104.62:44969 requested a full tablet report, sending...
I20250110 20:54:01.653460 8770 ts_manager.cc:194] Registered new tserver with Master: 871a805f7e704954af58e3aa4ee77dc8 (127.0.104.2:42789)
I20250110 20:54:01.654135 8770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:59303
W20250110 20:54:01.947170 9090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:01.947438 9090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:01.947546 9090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:01.954839 9090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:01.955049 9090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:54:01.963780 9090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44969
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:01.965229 9090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:01.966832 9090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:01.977173 9096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:01.977238 9098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:01.977259 9095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:01.978186 9090 server_base.cc:1034] running on GCE node
I20250110 20:54:01.978690 9090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:01.979506 9090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:01.980695 9090 hybrid_clock.cc:648] HybridClock initialized: now 1736542441980627 us; error 40 us; skew 500 ppm
I20250110 20:54:01.980950 9090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:01.982923 9090 webserver.cc:458] Webserver started at http://127.0.104.3:39865/ using document root <none> and password file <none>
I20250110 20:54:01.983579 9090 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:01.983678 9090 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:01.983943 9090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:01.986256 9090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.986953 9090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.991356 9090 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250110 20:54:01.994191 9104 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:01.995365 9090 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:54:01.995515 9090 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd"
format_stamp: "Formatted at 2025-01-10 20:54:01 on dist-test-slave-1h8z"
I20250110 20:54:01.995652 9090 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:02.008188 9090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:02.008821 9090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:02.009033 9090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:02.009776 9090 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:02.011099 9090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:02.011175 9090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:02.011258 9090 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:02.011299 9090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:02.038695 9090 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:33411
I20250110 20:54:02.038729 9216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:33411 every 8 connection(s)
I20250110 20:54:02.039899 9090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:54:02.043879 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9090
I20250110 20:54:02.044034 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:54:02.051245 9217 heartbeater.cc:346] Connected to a master server at 127.0.104.62:44969
I20250110 20:54:02.051474 9217 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:02.052140 9217 heartbeater.cc:510] Master 127.0.104.62:44969 requested a full tablet report, sending...
I20250110 20:54:02.053244 8770 ts_manager.cc:194] Registered new tserver with Master: e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411)
I20250110 20:54:02.053984 8770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:59903
I20250110 20:54:02.058581 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:54:02.071990 8770 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55088:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250110 20:54:02.077373 8770 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:54:02.101969 8891 tablet_service.cc:1467] Processing CreateTablet for tablet ac0b53ab700d44c49aa73e85d672fa3c (DEFAULT_TABLE table=TestTable [id=eda8ecdaac63492885331f55479b045c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:54:02.101969 9022 tablet_service.cc:1467] Processing CreateTablet for tablet ac0b53ab700d44c49aa73e85d672fa3c (DEFAULT_TABLE table=TestTable [id=eda8ecdaac63492885331f55479b045c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:54:02.102687 9152 tablet_service.cc:1467] Processing CreateTablet for tablet ac0b53ab700d44c49aa73e85d672fa3c (DEFAULT_TABLE table=TestTable [id=eda8ecdaac63492885331f55479b045c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:54:02.103439 9022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0b53ab700d44c49aa73e85d672fa3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:02.103451 8891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0b53ab700d44c49aa73e85d672fa3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:02.103927 9152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac0b53ab700d44c49aa73e85d672fa3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:02.113266 9235 tablet_bootstrap.cc:492] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd: Bootstrap starting.
I20250110 20:54:02.115774 9236 tablet_bootstrap.cc:492] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Bootstrap starting.
I20250110 20:54:02.117152 9237 tablet_bootstrap.cc:492] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Bootstrap starting.
I20250110 20:54:02.117762 9235 tablet_bootstrap.cc:654] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:02.119032 9236 tablet_bootstrap.cc:654] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:02.119040 9235 log.cc:826] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:02.119905 9237 tablet_bootstrap.cc:654] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:02.120105 9236 log.cc:826] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:02.120893 9237 log.cc:826] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:02.121629 9235 tablet_bootstrap.cc:492] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd: No bootstrap required, opened a new log
I20250110 20:54:02.121876 9235 ts_tablet_manager.cc:1397] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20250110 20:54:02.122674 9236 tablet_bootstrap.cc:492] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: No bootstrap required, opened a new log
I20250110 20:54:02.122915 9237 tablet_bootstrap.cc:492] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: No bootstrap required, opened a new log
I20250110 20:54:02.122953 9236 ts_tablet_manager.cc:1397] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20250110 20:54:02.123128 9237 ts_tablet_manager.cc:1397] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20250110 20:54:02.128767 9237 raft_consensus.cc:357] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.129069 9237 raft_consensus.cc:383] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:54:02.129175 9237 raft_consensus.cc:738] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da7d2e88cc74473ba3c7212ea6b5c9e8, State: Initialized, Role: FOLLOWER
I20250110 20:54:02.129707 9237 consensus_queue.cc:260] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.130622 8956 heartbeater.cc:502] Master 127.0.104.62:44969 was elected leader, sending a full tablet report...
I20250110 20:54:02.130940 9236 raft_consensus.cc:357] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.130916 9235 raft_consensus.cc:357] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.131189 9237 ts_tablet_manager.cc:1428] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Time spent starting tablet: real 0.008s user 0.003s sys 0.005s
I20250110 20:54:02.131286 9235 raft_consensus.cc:383] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:54:02.131286 9236 raft_consensus.cc:383] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:54:02.131376 9235 raft_consensus.cc:738] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9493f9f62bf4229ac45dfa0d41dc1dd, State: Initialized, Role: FOLLOWER
I20250110 20:54:02.131376 9236 raft_consensus.cc:738] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 871a805f7e704954af58e3aa4ee77dc8, State: Initialized, Role: FOLLOWER
I20250110 20:54:02.132018 9236 consensus_queue.cc:260] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.132018 9235 consensus_queue.cc:260] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.133419 9087 heartbeater.cc:502] Master 127.0.104.62:44969 was elected leader, sending a full tablet report...
I20250110 20:54:02.133476 9217 heartbeater.cc:502] Master 127.0.104.62:44969 was elected leader, sending a full tablet report...
I20250110 20:54:02.133877 9236 ts_tablet_manager.cc:1428] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20250110 20:54:02.135105 9235 ts_tablet_manager.cc:1428] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
W20250110 20:54:02.139760 9088 tablet.cc:2367] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:54:02.154927 9243 raft_consensus.cc:491] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:54:02.155113 9243 raft_consensus.cc:513] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.156522 9243 leader_election.cc:290] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da7d2e88cc74473ba3c7212ea6b5c9e8 (127.0.104.1:38959), e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411)
I20250110 20:54:02.164096 9172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "871a805f7e704954af58e3aa4ee77dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" is_pre_election: true
I20250110 20:54:02.164104 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "871a805f7e704954af58e3aa4ee77dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: true
I20250110 20:54:02.164525 8911 raft_consensus.cc:2463] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 871a805f7e704954af58e3aa4ee77dc8 in term 0.
I20250110 20:54:02.164536 9172 raft_consensus.cc:2463] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 871a805f7e704954af58e3aa4ee77dc8 in term 0.
I20250110 20:54:02.165103 8976 leader_election.cc:304] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 871a805f7e704954af58e3aa4ee77dc8, da7d2e88cc74473ba3c7212ea6b5c9e8; no voters:
I20250110 20:54:02.165488 9243 raft_consensus.cc:2798] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:54:02.165632 9243 raft_consensus.cc:491] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:54:02.165717 9243 raft_consensus.cc:3054] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:02.167743 9243 raft_consensus.cc:513] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.168254 9243 leader_election.cc:290] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [CANDIDATE]: Term 1 election: Requested vote from peers da7d2e88cc74473ba3c7212ea6b5c9e8 (127.0.104.1:38959), e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411)
I20250110 20:54:02.168603 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "871a805f7e704954af58e3aa4ee77dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8"
I20250110 20:54:02.168661 9172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "871a805f7e704954af58e3aa4ee77dc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd"
I20250110 20:54:02.168761 8911 raft_consensus.cc:3054] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:02.168819 9172 raft_consensus.cc:3054] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:02.170818 9172 raft_consensus.cc:2463] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871a805f7e704954af58e3aa4ee77dc8 in term 1.
I20250110 20:54:02.170837 8911 raft_consensus.cc:2463] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 871a805f7e704954af58e3aa4ee77dc8 in term 1.
I20250110 20:54:02.171186 8976 leader_election.cc:304] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 871a805f7e704954af58e3aa4ee77dc8, da7d2e88cc74473ba3c7212ea6b5c9e8; no voters:
I20250110 20:54:02.171445 9243 raft_consensus.cc:2798] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:02.171797 9243 raft_consensus.cc:695] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [term 1 LEADER]: Becoming Leader. State: Replica: 871a805f7e704954af58e3aa4ee77dc8, State: Running, Role: LEADER
I20250110 20:54:02.172252 9243 consensus_queue.cc:237] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } }
I20250110 20:54:02.176318 8769 catalog_manager.cc:5526] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 871a805f7e704954af58e3aa4ee77dc8 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "871a805f7e704954af58e3aa4ee77dc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "871a805f7e704954af58e3aa4ee77dc8" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 42789 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:02.188544 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:54:02.190313 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver da7d2e88cc74473ba3c7212ea6b5c9e8 to finish bootstrapping
I20250110 20:54:02.197466 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 871a805f7e704954af58e3aa4ee77dc8 to finish bootstrapping
I20250110 20:54:02.203469 416 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e9493f9f62bf4229ac45dfa0d41dc1dd to finish bootstrapping
I20250110 20:54:02.210244 8951 generic_service.cc:175] {username='slave'} at 127.0.0.1:36190 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20250110 20:54:02.211521 9082 generic_service.cc:175] {username='slave'} at 127.0.0.1:41024 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20250110 20:54:02.212672 9212 generic_service.cc:175] {username='slave'} at 127.0.0.1:56982 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20250110 20:54:02.226773 9172 raft_consensus.cc:1270] T ac0b53ab700d44c49aa73e85d672fa3c P e9493f9f62bf4229ac45dfa0d41dc1dd [term 1 FOLLOWER]: Refusing update from remote peer 871a805f7e704954af58e3aa4ee77dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:54:02.226819 8911 raft_consensus.cc:1270] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8 [term 1 FOLLOWER]: Refusing update from remote peer 871a805f7e704954af58e3aa4ee77dc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:54:02.227659 9243 consensus_queue.cc:1035] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:54:02.227967 9246 consensus_queue.cc:1035] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:54:02.230381 8957 tablet.cc:2367] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:54:02.239558 9250 mvcc.cc:204] Tried to move back new op lower bound from 7112877843352248320 to 7112877843141341184. Current Snapshot: MvccSnapshot[applied={T|T < 7112877843352248320}]
I20250110 20:54:02.261270 9082 generic_service.cc:175] {username='slave'} at 127.0.0.1:41024 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20250110 20:54:02.262193 9082 generic_service.cc:175] {username='slave'} at 127.0.0.1:41024 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20250110 20:54:02.263059 9082 generic_service.cc:175] {username='slave'} at 127.0.0.1:41024 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20250110 20:54:02.263900 8951 generic_service.cc:175] {username='slave'} at 127.0.0.1:36190 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20250110 20:54:02.264693 8951 generic_service.cc:175] {username='slave'} at 127.0.0.1:36190 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20250110 20:54:02.265421 8951 generic_service.cc:175] {username='slave'} at 127.0.0.1:36190 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20250110 20:54:02.265772 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9090
I20250110 20:54:02.266395 9212 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:02.348685 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9090
I20250110 20:54:02.365602 416 test_util.cc:274] Using random seed: 1265246842
I20250110 20:54:02.377362 416 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
W20250110 20:54:02.388263 9247 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250110 20:54:02.388911 8975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111)
W20250110 20:54:02.389966 9253 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Injecting 600ms of latency in SegmentAllocator::Sync()
W20250110 20:54:02.390281 8975 consensus_peers.cc:487] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 -> Peer e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411): Couldn't send request to peer e9493f9f62bf4229ac45dfa0d41dc1dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:54:02.428769 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.431505 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.483759 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.485715 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.537973 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.540042 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.592077 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.594002 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.646334 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.648429 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.700536 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.702593 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.754788 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.756891 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.809079 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.811204 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.863346 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.865386 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.917543 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.919513 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:02.971678 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:02.973754 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.026008 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.028100 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.080219 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.082252 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.134439 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.136482 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.188547 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.190543 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.242789 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.244824 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.296973 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.298970 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.351024 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.352964 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:03.382938 8975 consensus_peers.cc:487] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 -> Peer e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411): Couldn't send request to peer e9493f9f62bf4229ac45dfa0d41dc1dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250110 20:54:03.405108 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.407209 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.459353 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.461465 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.513870 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.515879 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.567906 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.569836 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.589978 9268 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41046 (ReqId={client: 8ef6d9e2de56440a92294c80365748c2, seq_no=0, attempt_no=0}) took 1203 ms. Trace:
I20250110 20:54:03.590111 9268 rpcz_store.cc:276] 0110 20:54:02.386098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:54:02.386157 (+ 59us) service_pool.cc:225] Handling call
0110 20:54:03.589929 (+1203772us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:54:02.387236 (+ 0us) write_op.cc:183] PREPARE: starting on tablet ac0b53ab700d44c49aa73e85d672fa3c
0110 20:54:02.387321 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:54:02.387326 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:54:02.387328 (+ 2us) tablet.cc:659] Decoding operations
0110 20:54:02.387400 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:54:02.387411 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:54:02.387414 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:54:02.387456 (+ 42us) tablet.cc:698] Row locks acquired
0110 20:54:02.387457 (+ 1us) write_op.cc:260] PREPARE: finished
0110 20:54:02.387570 (+ 113us) write_op.cc:270] Start()
0110 20:54:02.387597 (+ 27us) write_op.cc:276] Timestamp: P: 1736542442387528 usec, L: 0
0110 20:54:02.387599 (+ 2us) op_driver.cc:348] REPLICATION: starting
0110 20:54:02.387812 (+ 213us) log.cc:844] Serialized 345 byte log entry
0110 20:54:03.588743 (+1200931us) op_driver.cc:464] REPLICATION: finished
0110 20:54:03.589161 (+ 418us) write_op.cc:301] APPLY: starting
0110 20:54:03.589178 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:54:03.589213 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 20:54:03.589238 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:54:03.589330 (+ 92us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:54:03.589360 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:54:03.589365 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:54:03.589463 (+ 98us) log.cc:844] Serialized 22 byte log entry
0110 20:54:03.589563 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:54:03.589585 (+ 22us) write_op.cc:454] Released schema lock
0110 20:54:03.589592 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":316,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":681,"prepare.run_wall_time_us":681,"raft.queue_time_us":75,"raft.run_cpu_time_us":678,"raft.run_wall_time_us":678,"replication_time_us":1201111,"spinlock_wait_cycles":79744,"thread_start_us":224,"threads_started":1}]]}
I20250110 20:54:03.622042 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.624064 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.676187 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.678186 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.730458 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.732622 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.784878 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.787038 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:03.794364 9247 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250110 20:54:03.795935 9253 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250110 20:54:03.839241 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.841354 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.893505 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.895632 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:03.947845 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:03.949869 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.002187 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.004343 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.056583 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.058686 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.110870 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.112891 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.165043 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.167490 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.219790 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.221788 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.274029 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.276120 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.328254 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.330262 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:04.332775 8975 consensus_peers.cc:487] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 -> Peer e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411): Couldn't send request to peer e9493f9f62bf4229ac45dfa0d41dc1dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250110 20:54:04.382532 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.384507 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.436618 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.438536 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.490614 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.492591 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.544708 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.546679 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.598760 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.600733 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.652817 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.654840 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.707001 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.709007 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.761269 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.763381 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.815678 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.817888 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.870079 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.872198 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.924574 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.927381 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.980234 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:04.982267 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:04.995780 9274 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41046 (ReqId={client: 8ef6d9e2de56440a92294c80365748c2, seq_no=1, attempt_no=0}) took 1203 ms. Trace:
I20250110 20:54:04.995910 9274 rpcz_store.cc:276] 0110 20:54:03.792568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:54:03.792628 (+ 60us) service_pool.cc:225] Handling call
0110 20:54:04.995718 (+1203090us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:54:03.793602 (+ 0us) write_op.cc:183] PREPARE: starting on tablet ac0b53ab700d44c49aa73e85d672fa3c
0110 20:54:03.793710 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:54:03.793715 (+ 5us) write_op.cc:435] Acquired schema lock
0110 20:54:03.793717 (+ 2us) tablet.cc:659] Decoding operations
0110 20:54:03.793791 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:54:03.793802 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0110 20:54:03.793805 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:54:03.793854 (+ 49us) tablet.cc:698] Row locks acquired
0110 20:54:03.793856 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:54:03.793889 (+ 33us) write_op.cc:270] Start()
0110 20:54:03.793915 (+ 26us) write_op.cc:276] Timestamp: P: 1736542443793857 usec, L: 0
0110 20:54:03.793917 (+ 2us) op_driver.cc:348] REPLICATION: starting
0110 20:54:03.794074 (+ 157us) log.cc:844] Serialized 354 byte log entry
0110 20:54:04.994830 (+1200756us) op_driver.cc:464] REPLICATION: finished
0110 20:54:04.995190 (+ 360us) write_op.cc:301] APPLY: starting
0110 20:54:04.995212 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0110 20:54:04.995248 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0110 20:54:04.995273 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:54:04.995337 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:54:04.995362 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:54:04.995368 (+ 6us) write_op.cc:312] APPLY: finished
0110 20:54:04.995458 (+ 90us) log.cc:844] Serialized 22 byte log entry
0110 20:54:04.995581 (+ 123us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:54:04.995603 (+ 22us) write_op.cc:454] Released schema lock
0110 20:54:04.995611 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":296,"num_ops":1,"prepare.queue_time_us":314,"prepare.run_cpu_time_us":589,"prepare.run_wall_time_us":589,"raft.queue_time_us":79,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":216,"replication_time_us":1200879,"spinlock_wait_cycles":896,"thread_start_us":424,"threads_started":2}]]}
I20250110 20:54:05.034648 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.036825 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.089102 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.091209 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.143517 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.145601 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.197885 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.200079 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:05.200232 9247 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250110 20:54:05.201857 9253 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250110 20:54:05.252382 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.254611 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:05.274489 8975 consensus_peers.cc:487] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 -> Peer e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411): Couldn't send request to peer e9493f9f62bf4229ac45dfa0d41dc1dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250110 20:54:05.306825 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.308776 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.360996 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.363158 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.415385 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.417448 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.469622 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.471623 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.523700 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.525748 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.577849 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.579952 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.632227 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.634318 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.686626 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.688766 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.740849 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.743021 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.795272 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.797418 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.849704 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.851966 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.904152 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.906255 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:05.958592 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:05.960765 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.013119 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.015358 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.067581 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.069707 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.121994 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.124104 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.176389 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.178508 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.230823 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.233026 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:06.279610 8975 consensus_peers.cc:487] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 -> Peer e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411): Couldn't send request to peer e9493f9f62bf4229ac45dfa0d41dc1dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250110 20:54:06.285259 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.287745 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.340653 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.342886 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.395327 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.397516 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.401794 9280 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41046 (ReqId={client: 8ef6d9e2de56440a92294c80365748c2, seq_no=2, attempt_no=0}) took 1203 ms. Trace:
I20250110 20:54:06.401985 9280 rpcz_store.cc:276] 0110 20:54:05.198505 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:54:05.198572 (+ 67us) service_pool.cc:225] Handling call
0110 20:54:06.401716 (+1203144us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:54:05.199404 (+ 0us) write_op.cc:183] PREPARE: starting on tablet ac0b53ab700d44c49aa73e85d672fa3c
0110 20:54:05.199512 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:54:05.199518 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:54:05.199520 (+ 2us) tablet.cc:659] Decoding operations
0110 20:54:05.199598 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:54:05.199611 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0110 20:54:05.199615 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:54:05.199666 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:54:05.199668 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:54:05.199712 (+ 44us) write_op.cc:270] Start()
0110 20:54:05.199747 (+ 35us) write_op.cc:276] Timestamp: P: 1736542445199689 usec, L: 0
0110 20:54:05.199750 (+ 3us) op_driver.cc:348] REPLICATION: starting
0110 20:54:05.199914 (+ 164us) log.cc:844] Serialized 354 byte log entry
0110 20:54:06.400758 (+1200844us) op_driver.cc:464] REPLICATION: finished
0110 20:54:06.401164 (+ 406us) write_op.cc:301] APPLY: starting
0110 20:54:06.401182 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0110 20:54:06.401225 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0110 20:54:06.401245 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:54:06.401325 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:54:06.401353 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:54:06.401358 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:54:06.401451 (+ 93us) log.cc:844] Serialized 22 byte log entry
0110 20:54:06.401571 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:54:06.401593 (+ 22us) write_op.cc:454] Released schema lock
0110 20:54:06.401603 (+ 10us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":327,"num_ops":1,"prepare.queue_time_us":353,"prepare.run_cpu_time_us":669,"prepare.run_wall_time_us":670,"raft.queue_time_us":258,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":281,"replication_time_us":1200978,"thread_start_us":463,"threads_started":2}]]}
I20250110 20:54:06.449663 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.451653 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.503898 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.505937 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.558166 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.560263 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:06.605589 9247 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250110 20:54:06.607332 9253 log.cc:510] T ac0b53ab700d44c49aa73e85d672fa3c P da7d2e88cc74473ba3c7212ea6b5c9e8: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250110 20:54:06.612277 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.614231 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.666415 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.668885 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.721072 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.723140 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.775460 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.777567 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.829886 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.831954 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.884085 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.886133 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.938228 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.940256 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:06.992437 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:06.994448 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.046722 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.048782 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.100822 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.102775 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.154909 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.156944 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:07.196440 8975 consensus_peers.cc:487] T ac0b53ab700d44c49aa73e85d672fa3c P 871a805f7e704954af58e3aa4ee77dc8 -> Peer e9493f9f62bf4229ac45dfa0d41dc1dd (127.0.104.3:33411): Couldn't send request to peer e9493f9f62bf4229ac45dfa0d41dc1dd. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250110 20:54:07.209054 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.211045 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.263175 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.265134 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.317167 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.319092 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.371204 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.373210 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
W20250110 20:54:07.389317 8975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.3:33411: connect: Connection refused (error 111) [suppressed 25 similar messages]
I20250110 20:54:07.425345 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.427392 8911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.479548 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.481448 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.533579 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.535619 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.587510 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.589668 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.641747 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.643831 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.695950 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.698525 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.751040 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.752959 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.804956 9042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "871a805f7e704954af58e3aa4ee77dc8" is_pre_election: false
I20250110 20:54:07.807021 9286 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:41046 (ReqId={client: 8ef6d9e2de56440a92294c80365748c2, seq_no=3, attempt_no=0}) took 1203 ms. Trace:
I20250110 20:54:07.807301 8910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac0b53ab700d44c49aa73e85d672fa3c" candidate_uuid: "e9493f9f62bf4229ac45dfa0d41dc1dd" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "da7d2e88cc74473ba3c7212ea6b5c9e8" is_pre_election: false
I20250110 20:54:07.807224 9286 rpcz_store.cc:276] 0110 20:54:06.603834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:54:06.603903 (+ 69us) service_pool.cc:225] Handling call
0110 20:54:07.806955 (+1203052us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:54:06.604737 (+ 0us) write_op.cc:183] PREPARE: starting on tablet ac0b53ab700d44c49aa73e85d672fa3c
0110 20:54:06.604856 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:54:06.604862 (+ 6us) write_op.cc:435] Acquired schema lock
0110 20:54:06.604864 (+ 2us) tablet.cc:659] Decoding operations
0110 20:54:06.604942 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:54:06.604952 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0110 20:54:06.604955 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:54:06.605006 (+ 51us) tablet.cc:698] Row locks acquired
0110 20:54:06.605008 (+ 2us) write_op.cc:260] PREPARE: finished
0110 20:54:06.605044 (+ 36us) write_op.cc:270] Start()
0110 20:54:06.605073 (+ 29us) write_op.cc:276] Timestamp: P: 1736542446604983 usec, L: 0
0110 20:54:06.605088 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:54:06.605266 (+ 178us) log.cc:844] Serialized 354 byte log entry
0110 20:54:07.806044 (+1200778us) op_driver.cc:464] REPLICATION: finished
0110 20:54:07.806436 (+ 392us) write_op.cc:301] APPLY: starting
0110 20:54:07.806453 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0110 20:54:07.806488 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0110 20:54:07.806519 (+ 31us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:54:07.806590 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:54:07.806615 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:54:07.806620 (+ 5us) write_op.cc:312] APPLY: finished
0110 20:54:07.806690 (+ 70us) log.cc:844] Serialized 22 byte log entry
0110 20:54:07.806827 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:54:07.806849 (+ 22us) write_op.cc:454] Released schema lock
0110 20:54:07.806857 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":317,"num_ops":1,"prepare.queue_time_us":323,"prepare.run_cpu_time_us":657,"prepare.run_wall_time_us":658,"raft.queue_time_us":109,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":370,"replication_time_us":1200918,"spinlock_wait_cycles":3840,"thread_start_us":448,"threads_started":2}]]}
I20250110 20:54:08.011001 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8829
I20250110 20:54:08.011976 8951 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:08.095925 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8829
I20250110 20:54:08.109711 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8960
I20250110 20:54:08.110584 9082 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:08.197081 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8960
I20250110 20:54:08.210989 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 8739
I20250110 20:54:08.211933 8800 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:08.310190 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 8739
2025-01-10T20:54:08Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (8059 ms)
[----------] 12 tests from RaftConsensusElectionITest (188882 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20250110 20:54:08.344453 416 test_util.cc:274] Using random seed: 1271225697
I20250110 20:54:08.345983 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:54:08.346056 416 ts_itest-base.cc:113] --------------
I20250110 20:54:08.346093 416 ts_itest-base.cc:114] 1 tablet servers
I20250110 20:54:08.346143 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:54:08.346175 416 ts_itest-base.cc:116] --------------
2025-01-10T20:54:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:54:08Z Disabled control of system clock
I20250110 20:54:08.377472 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:45589
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:45589
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250110 20:54:08.679019 9300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:08.679288 9300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:08.679406 9300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:08.686694 9300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:54:08.686813 9300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:08.686882 9300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:54:08.686957 9300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:54:08.695673 9300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:45589
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:45589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:08.697055 9300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:08.698578 9300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:08.707574 9306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:08.707625 9308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:08.707650 9305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:08.708635 9300 server_base.cc:1034] running on GCE node
I20250110 20:54:08.709286 9300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:08.710460 9300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:08.711655 9300 hybrid_clock.cc:648] HybridClock initialized: now 1736542448711597 us; error 38 us; skew 500 ppm
I20250110 20:54:08.711938 9300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:08.713915 9300 webserver.cc:458] Webserver started at http://127.0.104.62:40229/ using document root <none> and password file <none>
I20250110 20:54:08.714537 9300 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:08.714646 9300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:08.714947 9300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:08.717298 9300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f437849c1cc14cc1bbac0eae7b07ebc0"
format_stamp: "Formatted at 2025-01-10 20:54:08 on dist-test-slave-1h8z"
I20250110 20:54:08.717967 9300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f437849c1cc14cc1bbac0eae7b07ebc0"
format_stamp: "Formatted at 2025-01-10 20:54:08 on dist-test-slave-1h8z"
I20250110 20:54:08.722451 9300 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20250110 20:54:08.725345 9314 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:08.726455 9300 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250110 20:54:08.726593 9300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f437849c1cc14cc1bbac0eae7b07ebc0"
format_stamp: "Formatted at 2025-01-10 20:54:08 on dist-test-slave-1h8z"
I20250110 20:54:08.726771 9300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:08.744802 9300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:08.745685 9300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:08.745976 9300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:08.761420 9300 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:45589
I20250110 20:54:08.761446 9365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:45589 every 8 connection(s)
I20250110 20:54:08.762562 9300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:54:08.763101 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9300
I20250110 20:54:08.763291 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:54:08.766471 9366 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:08.777983 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0: Bootstrap starting.
I20250110 20:54:08.780597 9366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:08.781419 9366 log.cc:826] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:08.783725 9366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0: No bootstrap required, opened a new log
I20250110 20:54:08.789415 9366 raft_consensus.cc:357] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45589 } }
I20250110 20:54:08.789681 9366 raft_consensus.cc:383] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:54:08.789760 9366 raft_consensus.cc:738] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f437849c1cc14cc1bbac0eae7b07ebc0, State: Initialized, Role: FOLLOWER
I20250110 20:54:08.790244 9366 consensus_queue.cc:260] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45589 } }
I20250110 20:54:08.790421 9366 raft_consensus.cc:397] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:54:08.790504 9366 raft_consensus.cc:491] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:54:08.790608 9366 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:08.792699 9366 raft_consensus.cc:513] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45589 } }
I20250110 20:54:08.793071 9366 leader_election.cc:304] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f437849c1cc14cc1bbac0eae7b07ebc0; no voters:
I20250110 20:54:08.793542 9366 leader_election.cc:290] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:08.793648 9371 raft_consensus.cc:2798] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:08.794183 9371 raft_consensus.cc:695] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [term 1 LEADER]: Becoming Leader. State: Replica: f437849c1cc14cc1bbac0eae7b07ebc0, State: Running, Role: LEADER
I20250110 20:54:08.794612 9371 consensus_queue.cc:237] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45589 } }
I20250110 20:54:08.795264 9366 sys_catalog.cc:564] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:54:08.796924 9372 sys_catalog.cc:455] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45589 } } }
I20250110 20:54:08.797165 9372 sys_catalog.cc:458] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [sys.catalog]: This master's current role is: LEADER
I20250110 20:54:08.797489 9373 sys_catalog.cc:455] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f437849c1cc14cc1bbac0eae7b07ebc0. Latest consensus state: current_term: 1 leader_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f437849c1cc14cc1bbac0eae7b07ebc0" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 45589 } } }
I20250110 20:54:08.797709 9373 sys_catalog.cc:458] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0 [sys.catalog]: This master's current role is: LEADER
I20250110 20:54:08.804003 9377 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:54:08.808390 9377 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:54:08.815299 9377 catalog_manager.cc:1348] Generated new cluster ID: 1a2fbd4381a9495d95d23ac3ece62859
I20250110 20:54:08.815392 9377 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:54:08.829865 9377 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:54:08.830958 9377 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:54:08.838701 9377 catalog_manager.cc:5899] T 00000000000000000000000000000000 P f437849c1cc14cc1bbac0eae7b07ebc0: Generated new TSK 0
I20250110 20:54:08.839623 9377 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:54:08.853895 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45589
--builtin_ntp_servers=127.0.104.20:43639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250110 20:54:09.163489 9390 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:54:09.163753 9390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:09.163833 9390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:09.163928 9390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:09.171214 9390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:09.171411 9390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:54:09.179944 9390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:45589
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:09.181299 9390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:09.182868 9390 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:09.193339 9395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:09.193375 9398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:09.193408 9396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:09.194342 9390 server_base.cc:1034] running on GCE node
I20250110 20:54:09.194830 9390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:09.195600 9390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:09.196777 9390 hybrid_clock.cc:648] HybridClock initialized: now 1736542449196737 us; error 35 us; skew 500 ppm
I20250110 20:54:09.197115 9390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:09.199115 9390 webserver.cc:458] Webserver started at http://127.0.104.1:45521/ using document root <none> and password file <none>
I20250110 20:54:09.199752 9390 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:09.199849 9390 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:09.200111 9390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:09.202370 9390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8b7ece86454241d6800fc338d3eab5e0"
format_stamp: "Formatted at 2025-01-10 20:54:09 on dist-test-slave-1h8z"
I20250110 20:54:09.203079 9390 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8b7ece86454241d6800fc338d3eab5e0"
format_stamp: "Formatted at 2025-01-10 20:54:09 on dist-test-slave-1h8z"
I20250110 20:54:09.207537 9390 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250110 20:54:09.210378 9404 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:09.211529 9390 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250110 20:54:09.211661 9390 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8b7ece86454241d6800fc338d3eab5e0"
format_stamp: "Formatted at 2025-01-10 20:54:09 on dist-test-slave-1h8z"
I20250110 20:54:09.211827 9390 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:09.223775 9390 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:09.224454 9390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:09.224699 9390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:09.225512 9390 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:09.226840 9390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:09.226930 9390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:09.227036 9390 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:09.227103 9390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:09.254503 9390 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37153
I20250110 20:54:09.254549 9516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37153 every 8 connection(s)
I20250110 20:54:09.255704 9390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:54:09.265511 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9390
I20250110 20:54:09.265686 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:54:09.267115 9517 heartbeater.cc:346] Connected to a master server at 127.0.104.62:45589
I20250110 20:54:09.267385 9517 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:09.268153 9517 heartbeater.cc:510] Master 127.0.104.62:45589 requested a full tablet report, sending...
I20250110 20:54:09.269861 9331 ts_manager.cc:194] Registered new tserver with Master: 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1:37153)
I20250110 20:54:09.271281 9331 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:56273
I20250110 20:54:09.280368 416 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250110 20:54:09.293701 416 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250110 20:54:09.293797 416 test_util.cc:274] Using random seed: 1272175047
I20250110 20:54:09.309221 9331 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54728:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:54:09.332981 9450 tablet_service.cc:1467] Processing CreateTablet for tablet 6c024e18cd3b437d88559f595c20744e (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250110 20:54:09.333017 9452 tablet_service.cc:1467] Processing CreateTablet for tablet 099efb3ab087470783c643625338c794 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250110 20:54:09.333057 9446 tablet_service.cc:1467] Processing CreateTablet for tablet a616652d32ac429996c93c4ac33329cc (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250110 20:54:09.334484 9450 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c024e18cd3b437d88559f595c20744e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.334872 9446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a616652d32ac429996c93c4ac33329cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.334910 9449 tablet_service.cc:1467] Processing CreateTablet for tablet 19ebd35c61ab4c2db0f9a812ddb6a8ab (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250110 20:54:09.335157 9452 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 099efb3ab087470783c643625338c794. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.335407 9449 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19ebd35c61ab4c2db0f9a812ddb6a8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.335569 9444 tablet_service.cc:1467] Processing CreateTablet for tablet 35ffb55942f94d3da8b77ce976d205c3 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250110 20:54:09.335959 9444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35ffb55942f94d3da8b77ce976d205c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.336666 9448 tablet_service.cc:1467] Processing CreateTablet for tablet f7d9631a1aff4b22a5192f87754b48b0 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250110 20:54:09.337118 9448 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d9631a1aff4b22a5192f87754b48b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.337689 9445 tablet_service.cc:1467] Processing CreateTablet for tablet eb0c80c947de459d816d19df7579b8f0 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250110 20:54:09.338116 9445 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb0c80c947de459d816d19df7579b8f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.338855 9447 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfcd2e4a9a343f9851df3763cf3c907 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250110 20:54:09.339298 9447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cfcd2e4a9a343f9851df3763cf3c907. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.335408 9443 tablet_service.cc:1467] Processing CreateTablet for tablet 2a8af2ef28b04f9b9939577815669448 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250110 20:54:09.343122 9443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a8af2ef28b04f9b9939577815669448. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.343966 9451 tablet_service.cc:1467] Processing CreateTablet for tablet cf40fb5b50ab4aae9a783923dfdfcd87 (DEFAULT_TABLE table=test-workload [id=f546f1ba2b93488dbf3e42424c2d9349]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250110 20:54:09.345849 9451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf40fb5b50ab4aae9a783923dfdfcd87. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:09.362929 9532 tablet_bootstrap.cc:492] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.366464 9532 tablet_bootstrap.cc:654] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.367396 9532 log.cc:826] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:09.369351 9532 tablet_bootstrap.cc:492] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.369556 9532 ts_tablet_manager.cc:1397] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:54:09.374997 9532 raft_consensus.cc:357] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.375279 9532 raft_consensus.cc:738] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.375640 9532 consensus_queue.cc:260] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.376480 9517 heartbeater.cc:502] Master 127.0.104.62:45589 was elected leader, sending a full tablet report...
I20250110 20:54:09.376974 9532 ts_tablet_manager.cc:1428] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250110 20:54:09.377312 9532 tablet_bootstrap.cc:492] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.378592 9532 tablet_bootstrap.cc:654] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.380141 9532 tablet_bootstrap.cc:492] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.380262 9532 ts_tablet_manager.cc:1397] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250110 20:54:09.380838 9532 raft_consensus.cc:357] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.381000 9532 raft_consensus.cc:738] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.381181 9532 consensus_queue.cc:260] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.381577 9532 ts_tablet_manager.cc:1428] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:09.381791 9532 tablet_bootstrap.cc:492] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.382967 9532 tablet_bootstrap.cc:654] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.384871 9532 tablet_bootstrap.cc:492] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.384991 9532 ts_tablet_manager.cc:1397] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 20:54:09.385506 9532 raft_consensus.cc:357] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.385638 9532 raft_consensus.cc:738] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.385777 9532 consensus_queue.cc:260] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.386123 9532 ts_tablet_manager.cc:1428] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:09.386329 9532 tablet_bootstrap.cc:492] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.387626 9532 tablet_bootstrap.cc:654] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.389153 9532 tablet_bootstrap.cc:492] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.389288 9532 ts_tablet_manager.cc:1397] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250110 20:54:09.389919 9532 raft_consensus.cc:357] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.390089 9532 raft_consensus.cc:738] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.390244 9532 consensus_queue.cc:260] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.390621 9532 ts_tablet_manager.cc:1428] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:09.390858 9532 tablet_bootstrap.cc:492] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.392112 9532 tablet_bootstrap.cc:654] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.393486 9532 tablet_bootstrap.cc:492] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.393611 9532 ts_tablet_manager.cc:1397] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20250110 20:54:09.394188 9532 raft_consensus.cc:357] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.394352 9532 raft_consensus.cc:738] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.394470 9532 consensus_queue.cc:260] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.394858 9532 ts_tablet_manager.cc:1428] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 20:54:09.395115 9532 tablet_bootstrap.cc:492] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.396256 9532 tablet_bootstrap.cc:654] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.397729 9532 tablet_bootstrap.cc:492] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.398222 9532 ts_tablet_manager.cc:1397] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 20:54:09.398707 9532 raft_consensus.cc:357] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.398872 9532 raft_consensus.cc:738] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.399031 9532 consensus_queue.cc:260] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.399453 9532 ts_tablet_manager.cc:1428] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 20:54:09.399667 9532 tablet_bootstrap.cc:492] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.401017 9532 tablet_bootstrap.cc:654] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.402467 9532 tablet_bootstrap.cc:492] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.402606 9532 ts_tablet_manager.cc:1397] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250110 20:54:09.403194 9532 raft_consensus.cc:357] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.403360 9532 raft_consensus.cc:738] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.403522 9532 consensus_queue.cc:260] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.403927 9532 ts_tablet_manager.cc:1428] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 20:54:09.404155 9532 tablet_bootstrap.cc:492] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.405586 9532 tablet_bootstrap.cc:654] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.407091 9532 tablet_bootstrap.cc:492] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.407231 9532 ts_tablet_manager.cc:1397] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250110 20:54:09.407725 9532 raft_consensus.cc:357] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.407866 9532 raft_consensus.cc:738] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.407977 9532 consensus_queue.cc:260] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.408308 9532 ts_tablet_manager.cc:1428] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:09.408591 9532 tablet_bootstrap.cc:492] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.409926 9532 tablet_bootstrap.cc:654] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.411370 9532 tablet_bootstrap.cc:492] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.411500 9532 ts_tablet_manager.cc:1397] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 20:54:09.412009 9532 raft_consensus.cc:357] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.412153 9532 raft_consensus.cc:738] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.412317 9532 consensus_queue.cc:260] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.412743 9532 ts_tablet_manager.cc:1428] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:09.412990 9532 tablet_bootstrap.cc:492] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: Bootstrap starting.
I20250110 20:54:09.414390 9532 tablet_bootstrap.cc:654] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:09.415938 9532 tablet_bootstrap.cc:492] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: No bootstrap required, opened a new log
I20250110 20:54:09.416087 9532 ts_tablet_manager.cc:1397] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 20:54:09.416644 9532 raft_consensus.cc:357] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.416765 9532 raft_consensus.cc:738] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Initialized, Role: FOLLOWER
I20250110 20:54:09.416922 9532 consensus_queue.cc:260] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.417281 9532 ts_tablet_manager.cc:1428] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:09.468211 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6c024e18cd3b437d88559f595c20744e"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.468451 9472 raft_consensus.cc:491] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.468600 9472 raft_consensus.cc:3054] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.470479 9472 raft_consensus.cc:513] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.470986 9472 leader_election.cc:304] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.471315 9472 leader_election.cc:290] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.471374 9534 raft_consensus.cc:2798] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.471843 9534 raft_consensus.cc:695] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.472313 9534 consensus_queue.cc:237] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.476459 9331 catalog_manager.cc:5526] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.483948 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb0c80c947de459d816d19df7579b8f0"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.484114 9472 raft_consensus.cc:491] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.484215 9472 raft_consensus.cc:3054] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.485867 9472 raft_consensus.cc:513] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.486068 9472 leader_election.cc:304] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.486220 9472 leader_election.cc:290] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.486276 9534 raft_consensus.cc:2798] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.486428 9534 raft_consensus.cc:695] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.486617 9534 consensus_queue.cc:237] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.488518 9331 catalog_manager.cc:5526] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.495990 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19ebd35c61ab4c2db0f9a812ddb6a8ab"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.496124 9472 raft_consensus.cc:491] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.496232 9472 raft_consensus.cc:3054] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.497831 9472 raft_consensus.cc:513] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.498019 9472 leader_election.cc:304] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.498167 9472 leader_election.cc:290] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.498245 9534 raft_consensus.cc:2798] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.498392 9534 raft_consensus.cc:695] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.498596 9534 consensus_queue.cc:237] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.500381 9331 catalog_manager.cc:5526] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
W20250110 20:54:09.507115 9518 tablet.cc:2367] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:54:09.508051 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf40fb5b50ab4aae9a783923dfdfcd87"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.508191 9472 raft_consensus.cc:491] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.508298 9472 raft_consensus.cc:3054] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.509905 9472 raft_consensus.cc:513] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.510100 9472 leader_election.cc:304] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.510244 9472 leader_election.cc:290] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.510299 9534 raft_consensus.cc:2798] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.510455 9534 raft_consensus.cc:695] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.510659 9534 consensus_queue.cc:237] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.512732 9331 catalog_manager.cc:5526] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.519693 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "35ffb55942f94d3da8b77ce976d205c3"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.519837 9472 raft_consensus.cc:491] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.519932 9472 raft_consensus.cc:3054] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.521544 9472 raft_consensus.cc:513] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.521729 9472 leader_election.cc:304] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.521875 9472 leader_election.cc:290] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.521971 9534 raft_consensus.cc:2798] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.522101 9534 raft_consensus.cc:695] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.522305 9534 consensus_queue.cc:237] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.524086 9331 catalog_manager.cc:5526] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.531387 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "099efb3ab087470783c643625338c794"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.531527 9472 raft_consensus.cc:491] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.531625 9472 raft_consensus.cc:3054] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.533216 9472 raft_consensus.cc:513] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.533404 9472 leader_election.cc:304] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.533555 9472 leader_election.cc:290] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.533627 9534 raft_consensus.cc:2798] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.533771 9534 raft_consensus.cc:695] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.533970 9534 consensus_queue.cc:237] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.535686 9331 catalog_manager.cc:5526] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.543278 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a8af2ef28b04f9b9939577815669448"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.543407 9472 raft_consensus.cc:491] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.543506 9472 raft_consensus.cc:3054] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.545070 9472 raft_consensus.cc:513] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.545281 9472 leader_election.cc:304] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.545449 9472 leader_election.cc:290] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.545536 9534 raft_consensus.cc:2798] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.545661 9534 raft_consensus.cc:695] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.545874 9534 consensus_queue.cc:237] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.547694 9331 catalog_manager.cc:5526] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.555109 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5cfcd2e4a9a343f9851df3763cf3c907"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.555251 9472 raft_consensus.cc:491] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.555344 9472 raft_consensus.cc:3054] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.556895 9472 raft_consensus.cc:513] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.557107 9472 leader_election.cc:304] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.557250 9472 leader_election.cc:290] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.557291 9534 raft_consensus.cc:2798] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.557452 9534 raft_consensus.cc:695] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.557646 9534 consensus_queue.cc:237] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.559347 9331 catalog_manager.cc:5526] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.567030 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f7d9631a1aff4b22a5192f87754b48b0"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.567170 9472 raft_consensus.cc:491] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.567274 9472 raft_consensus.cc:3054] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.568822 9472 raft_consensus.cc:513] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.569020 9472 leader_election.cc:304] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.569161 9472 leader_election.cc:290] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.569243 9534 raft_consensus.cc:2798] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.569391 9534 raft_consensus.cc:695] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.569595 9534 consensus_queue.cc:237] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.571421 9331 catalog_manager.cc:5526] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.578724 9472 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a616652d32ac429996c93c4ac33329cc"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.578902 9472 raft_consensus.cc:491] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:09.579002 9472 raft_consensus.cc:3054] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:09.580590 9472 raft_consensus.cc:513] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.580780 9472 leader_election.cc:304] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b7ece86454241d6800fc338d3eab5e0; no voters:
I20250110 20:54:09.580912 9472 leader_election.cc:290] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:09.580979 9534 raft_consensus.cc:2798] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:09.581135 9534 raft_consensus.cc:695] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.581317 9534 consensus_queue.cc:237] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.583096 9331 catalog_manager.cc:5526] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b7ece86454241d6800fc338d3eab5e0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "8b7ece86454241d6800fc338d3eab5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } health_report { overall_health: HEALTHY } } }
I20250110 20:54:09.590821 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c024e18cd3b437d88559f595c20744e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.591463 9558 tablet_replica.cc:331] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.591820 9558 raft_consensus.cc:2238] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:09.592100 9558 raft_consensus.cc:2267] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.593343 9558 ts_tablet_manager.cc:1905] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.597445 9558 ts_tablet_manager.cc:1918] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.597568 9558 log.cc:1198] T 6c024e18cd3b437d88559f595c20744e P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/6c024e18cd3b437d88559f595c20744e
I20250110 20:54:09.605768 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet eb0c80c947de459d816d19df7579b8f0 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.606000 9558 tablet_replica.cc:331] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.606148 9558 raft_consensus.cc:2238] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:09.606304 9558 raft_consensus.cc:2267] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.606993 9558 ts_tablet_manager.cc:1905] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.610531 9558 ts_tablet_manager.cc:1918] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.610632 9558 log.cc:1198] T eb0c80c947de459d816d19df7579b8f0 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/eb0c80c947de459d816d19df7579b8f0
I20250110 20:54:09.618629 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 19ebd35c61ab4c2db0f9a812ddb6a8ab with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.618865 9558 tablet_replica.cc:331] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.619041 9558 raft_consensus.cc:2238] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:09.619202 9558 raft_consensus.cc:2267] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.619880 9558 ts_tablet_manager.cc:1905] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.623414 9558 ts_tablet_manager.cc:1918] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.623507 9558 log.cc:1198] T 19ebd35c61ab4c2db0f9a812ddb6a8ab P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/19ebd35c61ab4c2db0f9a812ddb6a8ab
I20250110 20:54:09.631428 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet cf40fb5b50ab4aae9a783923dfdfcd87 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.631619 9558 tablet_replica.cc:331] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.631736 9558 raft_consensus.cc:2238] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:09.631860 9558 raft_consensus.cc:2267] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.632468 9558 ts_tablet_manager.cc:1905] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.635958 9558 ts_tablet_manager.cc:1918] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.636070 9558 log.cc:1198] T cf40fb5b50ab4aae9a783923dfdfcd87 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf40fb5b50ab4aae9a783923dfdfcd87
I20250110 20:54:09.643873 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ffb55942f94d3da8b77ce976d205c3 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.644080 9558 tablet_replica.cc:331] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.644193 9558 raft_consensus.cc:2238] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:09.644327 9558 raft_consensus.cc:2267] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.644925 9558 ts_tablet_manager.cc:1905] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.648422 9558 ts_tablet_manager.cc:1918] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.648504 9558 log.cc:1198] T 35ffb55942f94d3da8b77ce976d205c3 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/35ffb55942f94d3da8b77ce976d205c3
I20250110 20:54:09.656490 9472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "099efb3ab087470783c643625338c794"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.656656 9472 raft_consensus.cc:589] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Received request to step down
I20250110 20:54:09.656730 9472 raft_consensus.cc:3049] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:09.656780 9472 raft_consensus.cc:738] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.656949 9472 consensus_queue.cc:260] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.657066 9472 raft_consensus.cc:3054] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:09.664891 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 099efb3ab087470783c643625338c794 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.665095 9558 tablet_replica.cc:331] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.665215 9558 raft_consensus.cc:2238] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:09.665342 9558 raft_consensus.cc:2267] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.665962 9558 ts_tablet_manager.cc:1905] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.669456 9558 ts_tablet_manager.cc:1918] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.669543 9558 log.cc:1198] T 099efb3ab087470783c643625338c794 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/099efb3ab087470783c643625338c794
I20250110 20:54:09.677723 9472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2a8af2ef28b04f9b9939577815669448"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.677865 9472 raft_consensus.cc:589] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Received request to step down
I20250110 20:54:09.677938 9472 raft_consensus.cc:3049] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:09.677984 9472 raft_consensus.cc:738] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.678140 9472 consensus_queue.cc:260] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.678242 9472 raft_consensus.cc:3054] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:09.685998 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a8af2ef28b04f9b9939577815669448 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.686236 9558 tablet_replica.cc:331] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.686363 9558 raft_consensus.cc:2238] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:09.686491 9558 raft_consensus.cc:2267] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.687211 9558 ts_tablet_manager.cc:1905] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.690842 9558 ts_tablet_manager.cc:1918] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.690936 9558 log.cc:1198] T 2a8af2ef28b04f9b9939577815669448 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a8af2ef28b04f9b9939577815669448
I20250110 20:54:09.699085 9472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5cfcd2e4a9a343f9851df3763cf3c907"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.699229 9472 raft_consensus.cc:589] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Received request to step down
I20250110 20:54:09.699308 9472 raft_consensus.cc:3049] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:09.699354 9472 raft_consensus.cc:738] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.699499 9472 consensus_queue.cc:260] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.699604 9472 raft_consensus.cc:3054] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:09.707732 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfcd2e4a9a343f9851df3763cf3c907 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.707944 9558 tablet_replica.cc:331] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.708068 9558 raft_consensus.cc:2238] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:09.708197 9558 raft_consensus.cc:2267] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.708845 9558 ts_tablet_manager.cc:1905] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.712481 9558 ts_tablet_manager.cc:1918] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.712566 9558 log.cc:1198] T 5cfcd2e4a9a343f9851df3763cf3c907 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/5cfcd2e4a9a343f9851df3763cf3c907
I20250110 20:54:09.720535 9472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f7d9631a1aff4b22a5192f87754b48b0"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.720672 9472 raft_consensus.cc:589] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Received request to step down
I20250110 20:54:09.720742 9472 raft_consensus.cc:3049] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:09.720788 9472 raft_consensus.cc:738] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.720934 9472 consensus_queue.cc:260] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.721035 9472 raft_consensus.cc:3054] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:09.728773 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet f7d9631a1aff4b22a5192f87754b48b0 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.729000 9558 tablet_replica.cc:331] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.729132 9558 raft_consensus.cc:2238] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:09.729266 9558 raft_consensus.cc:2267] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.730019 9558 ts_tablet_manager.cc:1905] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.733909 9558 ts_tablet_manager.cc:1918] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.734014 9558 log.cc:1198] T f7d9631a1aff4b22a5192f87754b48b0 P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/f7d9631a1aff4b22a5192f87754b48b0
I20250110 20:54:09.742132 9472 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a616652d32ac429996c93c4ac33329cc"
dest_uuid: "8b7ece86454241d6800fc338d3eab5e0"
from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.742293 9472 raft_consensus.cc:589] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Received request to step down
I20250110 20:54:09.742383 9472 raft_consensus.cc:3049] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:09.742447 9472 raft_consensus.cc:738] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8b7ece86454241d6800fc338d3eab5e0, State: Running, Role: LEADER
I20250110 20:54:09.742638 9472 consensus_queue.cc:260] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b7ece86454241d6800fc338d3eab5e0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37153 } }
I20250110 20:54:09.742807 9472 raft_consensus.cc:3054] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:09.751670 9450 tablet_service.cc:1514] Processing DeleteTablet for tablet a616652d32ac429996c93c4ac33329cc with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:50452
I20250110 20:54:09.751905 9558 tablet_replica.cc:331] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: stopping tablet replica
I20250110 20:54:09.752055 9558 raft_consensus.cc:2238] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:09.752223 9558 raft_consensus.cc:2267] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:09.752974 9558 ts_tablet_manager.cc:1905] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:09.756775 9558 ts_tablet_manager.cc:1918] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:09.756873 9558 log.cc:1198] T a616652d32ac429996c93c4ac33329cc P 8b7ece86454241d6800fc338d3eab5e0: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/wals/a616652d32ac429996c93c4ac33329cc
I20250110 20:54:09.766208 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9390
I20250110 20:54:09.772631 9512 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:09.882284 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9390
I20250110 20:54:09.897497 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9300
I20250110 20:54:09.898486 9361 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:09.997591 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9300
2025-01-10T20:54:10Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (1685 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20250110 20:54:10.030387 416 test_util.cc:274] Using random seed: 1272911645
I20250110 20:54:10.031950 416 ts_itest-base.cc:112] Starting cluster with:
I20250110 20:54:10.032009 416 ts_itest-base.cc:113] --------------
I20250110 20:54:10.032055 416 ts_itest-base.cc:114] 3 tablet servers
I20250110 20:54:10.032104 416 ts_itest-base.cc:115] 3 replicas per TS
I20250110 20:54:10.032160 416 ts_itest-base.cc:116] --------------
2025-01-10T20:54:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:54:10Z Disabled control of system clock
I20250110 20:54:10.064455 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40227
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40227
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250110 20:54:10.363981 9569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:10.364217 9569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:10.364336 9569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:10.371610 9569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:54:10.371703 9569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:10.371767 9569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:54:10.371840 9569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:54:10.380462 9569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40227
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:10.381883 9569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:10.383487 9569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:10.392529 9575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:10.392694 9574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:10.392733 9577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:10.393441 9569 server_base.cc:1034] running on GCE node
I20250110 20:54:10.394090 9569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:10.395298 9569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:10.396499 9569 hybrid_clock.cc:648] HybridClock initialized: now 1736542450396450 us; error 26 us; skew 500 ppm
I20250110 20:54:10.396766 9569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:10.398728 9569 webserver.cc:458] Webserver started at http://127.0.104.62:38753/ using document root <none> and password file <none>
I20250110 20:54:10.399384 9569 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:10.399492 9569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:10.399758 9569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:10.402089 9569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8b41012a0d7748cbbe4818108f22f3eb"
format_stamp: "Formatted at 2025-01-10 20:54:10 on dist-test-slave-1h8z"
I20250110 20:54:10.402802 9569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8b41012a0d7748cbbe4818108f22f3eb"
format_stamp: "Formatted at 2025-01-10 20:54:10 on dist-test-slave-1h8z"
I20250110 20:54:10.407125 9569 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20250110 20:54:10.409863 9583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:10.410995 9569 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:54:10.411130 9569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8b41012a0d7748cbbe4818108f22f3eb"
format_stamp: "Formatted at 2025-01-10 20:54:10 on dist-test-slave-1h8z"
I20250110 20:54:10.411303 9569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:10.422325 9569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:10.422981 9569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:10.423221 9569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:10.438354 9569 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40227
I20250110 20:54:10.438388 9634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40227 every 8 connection(s)
I20250110 20:54:10.439478 9569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250110 20:54:10.441972 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9569
I20250110 20:54:10.442123 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250110 20:54:10.442961 9635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:10.453248 9635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb: Bootstrap starting.
I20250110 20:54:10.456080 9635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:10.457049 9635 log.cc:826] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:10.459241 9635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb: No bootstrap required, opened a new log
I20250110 20:54:10.464833 9635 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41012a0d7748cbbe4818108f22f3eb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40227 } }
I20250110 20:54:10.465124 9635 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:54:10.465194 9635 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b41012a0d7748cbbe4818108f22f3eb, State: Initialized, Role: FOLLOWER
I20250110 20:54:10.465644 9635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41012a0d7748cbbe4818108f22f3eb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40227 } }
I20250110 20:54:10.465809 9635 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250110 20:54:10.465894 9635 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250110 20:54:10.466004 9635 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:10.468015 9635 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41012a0d7748cbbe4818108f22f3eb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40227 } }
I20250110 20:54:10.468405 9635 leader_election.cc:304] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b41012a0d7748cbbe4818108f22f3eb; no voters:
I20250110 20:54:10.468863 9635 leader_election.cc:290] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250110 20:54:10.468983 9640 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:10.469527 9640 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [term 1 LEADER]: Becoming Leader. State: Replica: 8b41012a0d7748cbbe4818108f22f3eb, State: Running, Role: LEADER
I20250110 20:54:10.469959 9640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41012a0d7748cbbe4818108f22f3eb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40227 } }
I20250110 20:54:10.470510 9635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:54:10.472399 9641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b41012a0d7748cbbe4818108f22f3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41012a0d7748cbbe4818108f22f3eb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40227 } } }
I20250110 20:54:10.472476 9642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b41012a0d7748cbbe4818108f22f3eb. Latest consensus state: current_term: 1 leader_uuid: "8b41012a0d7748cbbe4818108f22f3eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b41012a0d7748cbbe4818108f22f3eb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40227 } } }
I20250110 20:54:10.472710 9641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [sys.catalog]: This master's current role is: LEADER
I20250110 20:54:10.472729 9642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb [sys.catalog]: This master's current role is: LEADER
I20250110 20:54:10.474303 9648 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:54:10.479444 9648 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:54:10.486793 9648 catalog_manager.cc:1348] Generated new cluster ID: 29d96bfeef724931b771ae5cae6984f2
I20250110 20:54:10.486886 9648 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:54:10.507714 9648 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:54:10.508759 9648 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:54:10.516844 9648 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 8b41012a0d7748cbbe4818108f22f3eb: Generated new TSK 0
I20250110 20:54:10.517486 9648 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:54:10.528007 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40227
--builtin_ntp_servers=127.0.104.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250110 20:54:10.839037 9659 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:54:10.839293 9659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:10.839382 9659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:10.839475 9659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:10.847059 9659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:10.847234 9659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250110 20:54:10.855656 9659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:10.856930 9659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:10.858480 9659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:10.868553 9667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:10.868575 9664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:10.868609 9665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:10.869611 9659 server_base.cc:1034] running on GCE node
I20250110 20:54:10.870086 9659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:10.870867 9659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:10.872049 9659 hybrid_clock.cc:648] HybridClock initialized: now 1736542450872008 us; error 40 us; skew 500 ppm
I20250110 20:54:10.872326 9659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:10.874197 9659 webserver.cc:458] Webserver started at http://127.0.104.1:36901/ using document root <none> and password file <none>
I20250110 20:54:10.874877 9659 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:10.874974 9659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:10.875278 9659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:10.877683 9659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7e59427cf93f4bd480e35b4ba3207e16"
format_stamp: "Formatted at 2025-01-10 20:54:10 on dist-test-slave-1h8z"
I20250110 20:54:10.878340 9659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7e59427cf93f4bd480e35b4ba3207e16"
format_stamp: "Formatted at 2025-01-10 20:54:10 on dist-test-slave-1h8z"
I20250110 20:54:10.882789 9659 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20250110 20:54:10.885604 9673 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:10.886716 9659 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250110 20:54:10.886900 9659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7e59427cf93f4bd480e35b4ba3207e16"
format_stamp: "Formatted at 2025-01-10 20:54:10 on dist-test-slave-1h8z"
I20250110 20:54:10.887076 9659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:10.905296 9659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:10.905918 9659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:10.906131 9659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:10.906929 9659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:10.908231 9659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:10.908310 9659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:10.908385 9659 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:10.908438 9659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:10.939296 9659 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:41893
I20250110 20:54:10.939365 9785 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:41893 every 8 connection(s)
I20250110 20:54:10.940425 9659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250110 20:54:10.940775 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9659
I20250110 20:54:10.940927 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250110 20:54:10.943951 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40227
--builtin_ntp_servers=127.0.104.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:54:10.952507 9786 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40227
I20250110 20:54:10.952764 9786 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:10.953508 9786 heartbeater.cc:510] Master 127.0.104.62:40227 requested a full tablet report, sending...
I20250110 20:54:10.955415 9600 ts_manager.cc:194] Registered new tserver with Master: 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893)
I20250110 20:54:10.956862 9600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:49507
W20250110 20:54:11.250032 9789 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:54:11.250289 9789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:11.250367 9789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:11.250471 9789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:11.257613 9789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:11.257795 9789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250110 20:54:11.266453 9789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:11.267808 9789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:11.269327 9789 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:11.279409 9794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:11.279462 9795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:11.279503 9797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:11.280630 9789 server_base.cc:1034] running on GCE node
I20250110 20:54:11.281085 9789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:11.281831 9789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:11.283003 9789 hybrid_clock.cc:648] HybridClock initialized: now 1736542451282958 us; error 40 us; skew 500 ppm
I20250110 20:54:11.283242 9789 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:11.289052 9789 webserver.cc:458] Webserver started at http://127.0.104.2:34363/ using document root <none> and password file <none>
I20250110 20:54:11.289667 9789 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:11.289750 9789 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:11.289987 9789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:11.292310 9789 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
format_stamp: "Formatted at 2025-01-10 20:54:11 on dist-test-slave-1h8z"
I20250110 20:54:11.292943 9789 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
format_stamp: "Formatted at 2025-01-10 20:54:11 on dist-test-slave-1h8z"
I20250110 20:54:11.297253 9789 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20250110 20:54:11.300061 9803 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:11.301127 9789 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250110 20:54:11.301254 9789 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
format_stamp: "Formatted at 2025-01-10 20:54:11 on dist-test-slave-1h8z"
I20250110 20:54:11.301401 9789 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:11.314249 9789 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:11.314877 9789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:11.315073 9789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:11.315796 9789 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:11.317059 9789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:11.317134 9789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:11.317204 9789 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:11.317265 9789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:11.344326 9789 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35491
I20250110 20:54:11.344381 9915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35491 every 8 connection(s)
I20250110 20:54:11.345446 9789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250110 20:54:11.346410 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9789
I20250110 20:54:11.346556 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250110 20:54:11.349758 416 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskrkqplS/build/asan/bin/kudu
/tmp/dist-test-taskrkqplS/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40227
--builtin_ntp_servers=127.0.104.20:44097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250110 20:54:11.357766 9916 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40227
I20250110 20:54:11.358034 9916 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:11.358881 9916 heartbeater.cc:510] Master 127.0.104.62:40227 requested a full tablet report, sending...
I20250110 20:54:11.360270 9600 ts_manager.cc:194] Registered new tserver with Master: 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2:35491)
I20250110 20:54:11.361188 9600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:52827
W20250110 20:54:11.651834 9919 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250110 20:54:11.652099 9919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:54:11.652185 9919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:54:11.652276 9919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:54:11.659543 9919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:54:11.659739 9919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250110 20:54:11.668469 9919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40227
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:24 UTC on 24a791456cd2
build id 4115
ASAN enabled
I20250110 20:54:11.669734 9919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:54:11.671281 9919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:54:11.681681 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:11.681663 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:54:11.681682 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:54:11.682425 9919 server_base.cc:1034] running on GCE node
I20250110 20:54:11.682924 9919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:54:11.683727 9919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:54:11.684912 9919 hybrid_clock.cc:648] HybridClock initialized: now 1736542451684876 us; error 28 us; skew 500 ppm
I20250110 20:54:11.685184 9919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:54:11.687130 9919 webserver.cc:458] Webserver started at http://127.0.104.3:36617/ using document root <none> and password file <none>
I20250110 20:54:11.687764 9919 fs_manager.cc:362] Metadata directory not provided
I20250110 20:54:11.687858 9919 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:54:11.688144 9919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:54:11.690418 9919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "23c6369336864e6a88a75d229d4448b6"
format_stamp: "Formatted at 2025-01-10 20:54:11 on dist-test-slave-1h8z"
I20250110 20:54:11.691125 9919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "23c6369336864e6a88a75d229d4448b6"
format_stamp: "Formatted at 2025-01-10 20:54:11 on dist-test-slave-1h8z"
I20250110 20:54:11.695538 9919 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20250110 20:54:11.698400 9933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:11.699538 9919 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250110 20:54:11.699671 9919 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "23c6369336864e6a88a75d229d4448b6"
format_stamp: "Formatted at 2025-01-10 20:54:11 on dist-test-slave-1h8z"
I20250110 20:54:11.699844 9919 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:54:11.716181 9919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:54:11.716905 9919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:54:11.717146 9919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:54:11.717975 9919 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:54:11.719336 9919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:54:11.719424 9919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:11.719523 9919 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:54:11.719588 9919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:54:11.746886 9919 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:35289
I20250110 20:54:11.746941 10045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:35289 every 8 connection(s)
I20250110 20:54:11.748042 9919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250110 20:54:11.753546 416 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskrkqplS/build/asan/bin/kudu as pid 9919
I20250110 20:54:11.753718 416 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250110 20:54:11.759902 10046 heartbeater.cc:346] Connected to a master server at 127.0.104.62:40227
I20250110 20:54:11.760154 10046 heartbeater.cc:463] Registering TS with master...
I20250110 20:54:11.760833 10046 heartbeater.cc:510] Master 127.0.104.62:40227 requested a full tablet report, sending...
I20250110 20:54:11.761940 9600 ts_manager.cc:194] Registered new tserver with Master: 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:11.762650 9600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:49057
I20250110 20:54:11.768343 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:54:11.780830 416 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:54:11.780902 416 test_util.cc:274] Using random seed: 1274662170
I20250110 20:54:11.795450 9600 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49076:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:54:11.797183 9600 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:54:11.841135 9845 tablet_service.cc:1467] Processing CreateTablet for tablet 105ab07a05d8485297bc4997e40875d6 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250110 20:54:11.840601 9843 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddff8ac3c8a433a8587a9a657000089 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250110 20:54:11.841817 9849 tablet_service.cc:1467] Processing CreateTablet for tablet 77ec0f41406a451e94a103173bed7483 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250110 20:54:11.842008 9846 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb8ed70417453aa19acb9c5125a62e (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250110 20:54:11.841320 9850 tablet_service.cc:1467] Processing CreateTablet for tablet 56acdd81451e447ebb4bf1943947f52c (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250110 20:54:11.842501 9848 tablet_service.cc:1467] Processing CreateTablet for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250110 20:54:11.842471 9847 tablet_service.cc:1467] Processing CreateTablet for tablet a50e6ed61c574319866dbcadd6c81632 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250110 20:54:11.842715 9843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ddff8ac3c8a433a8587a9a657000089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.840713 9842 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0d07a127584348bbbaf0706ab35c97 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250110 20:54:11.840669 9844 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250110 20:54:11.840076 9851 tablet_service.cc:1467] Processing CreateTablet for tablet 4b110d55d2e34081844cb4797d97e289 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250110 20:54:11.843256 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.843765 9848 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.844084 9849 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ec0f41406a451e94a103173bed7483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.845119 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 105ab07a05d8485297bc4997e40875d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.845636 9847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50e6ed61c574319866dbcadd6c81632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.846144 9842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0d07a127584348bbbaf0706ab35c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.845693 9720 tablet_service.cc:1467] Processing CreateTablet for tablet 56acdd81451e447ebb4bf1943947f52c (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250110 20:54:11.846771 9846 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fb8ed70417453aa19acb9c5125a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.845743 9721 tablet_service.cc:1467] Processing CreateTablet for tablet 4b110d55d2e34081844cb4797d97e289 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250110 20:54:11.846954 9850 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56acdd81451e447ebb4bf1943947f52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.847384 9720 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56acdd81451e447ebb4bf1943947f52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.849781 9851 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b110d55d2e34081844cb4797d97e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.847993 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 77ec0f41406a451e94a103173bed7483 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250110 20:54:11.850170 9717 tablet_service.cc:1467] Processing CreateTablet for tablet a50e6ed61c574319866dbcadd6c81632 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250110 20:54:11.849089 9718 tablet_service.cc:1467] Processing CreateTablet for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250110 20:54:11.849757 9716 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb8ed70417453aa19acb9c5125a62e (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250110 20:54:11.850385 9981 tablet_service.cc:1467] Processing CreateTablet for tablet 4b110d55d2e34081844cb4797d97e289 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250110 20:54:11.851367 9715 tablet_service.cc:1467] Processing CreateTablet for tablet 105ab07a05d8485297bc4997e40875d6 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250110 20:54:11.851377 9975 tablet_service.cc:1467] Processing CreateTablet for tablet 105ab07a05d8485297bc4997e40875d6 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250110 20:54:11.852001 9981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b110d55d2e34081844cb4797d97e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.852492 9716 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fb8ed70417453aa19acb9c5125a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.852675 9719 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ec0f41406a451e94a103173bed7483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.852854 9715 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 105ab07a05d8485297bc4997e40875d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.853003 9718 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.853394 9714 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250110 20:54:11.853771 9714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.854223 9979 tablet_service.cc:1467] Processing CreateTablet for tablet 77ec0f41406a451e94a103173bed7483 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250110 20:54:11.854676 9979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ec0f41406a451e94a103173bed7483. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.854938 9977 tablet_service.cc:1467] Processing CreateTablet for tablet a50e6ed61c574319866dbcadd6c81632 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250110 20:54:11.855340 9977 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50e6ed61c574319866dbcadd6c81632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.855562 9713 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddff8ac3c8a433a8587a9a657000089 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250110 20:54:11.850921 9973 tablet_service.cc:1467] Processing CreateTablet for tablet 8ddff8ac3c8a433a8587a9a657000089 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250110 20:54:11.856010 9713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ddff8ac3c8a433a8587a9a657000089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.856143 9973 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ddff8ac3c8a433a8587a9a657000089. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.856791 9978 tablet_service.cc:1467] Processing CreateTablet for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250110 20:54:11.857224 9978 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.857926 9980 tablet_service.cc:1467] Processing CreateTablet for tablet 56acdd81451e447ebb4bf1943947f52c (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250110 20:54:11.858377 9980 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56acdd81451e447ebb4bf1943947f52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.858584 9976 tablet_service.cc:1467] Processing CreateTablet for tablet f0fb8ed70417453aa19acb9c5125a62e (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250110 20:54:11.859073 9976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0fb8ed70417453aa19acb9c5125a62e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.859392 9712 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0d07a127584348bbbaf0706ab35c97 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250110 20:54:11.859781 9712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0d07a127584348bbbaf0706ab35c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.858350 9974 tablet_service.cc:1467] Processing CreateTablet for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250110 20:54:11.860631 9717 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50e6ed61c574319866dbcadd6c81632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.860864 9974 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.864352 9975 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 105ab07a05d8485297bc4997e40875d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.872153 9721 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b110d55d2e34081844cb4797d97e289. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.907881 10071 tablet_bootstrap.cc:492] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:11.906543 9972 tablet_service.cc:1467] Processing CreateTablet for tablet 6d0d07a127584348bbbaf0706ab35c97 (DEFAULT_TABLE table=test-workload [id=fce4dc2be01d4a4a9aaec4f86a4f51ef]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250110 20:54:11.911692 10071 tablet_bootstrap.cc:654] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.913038 10071 log.cc:826] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:11.918951 9972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d0d07a127584348bbbaf0706ab35c97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:54:11.921787 10071 tablet_bootstrap.cc:492] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:11.922122 10071 ts_tablet_manager.cc:1397] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20250110 20:54:11.930269 10071 raft_consensus.cc:357] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.930838 10071 raft_consensus.cc:738] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.931442 10071 consensus_queue.cc:260] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.933398 10070 tablet_bootstrap.cc:492] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:11.938585 10070 tablet_bootstrap.cc:654] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.939287 10069 tablet_bootstrap.cc:492] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:11.944378 10070 log.cc:826] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:11.944792 10069 tablet_bootstrap.cc:654] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.945555 10071 ts_tablet_manager.cc:1428] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.023s user 0.012s sys 0.004s
I20250110 20:54:11.945922 9786 heartbeater.cc:502] Master 127.0.104.62:40227 was elected leader, sending a full tablet report...
I20250110 20:54:11.946046 10069 log.cc:826] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Log is configured to *not* fsync() on all Append() calls
I20250110 20:54:11.946671 10071 tablet_bootstrap.cc:492] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:11.948066 10070 tablet_bootstrap.cc:492] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:11.948348 10070 ts_tablet_manager.cc:1397] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250110 20:54:11.948357 10071 tablet_bootstrap.cc:654] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.955003 10069 tablet_bootstrap.cc:492] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:11.955344 10069 ts_tablet_manager.cc:1397] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.006s
I20250110 20:54:11.957130 10070 raft_consensus.cc:357] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.957608 10070 raft_consensus.cc:738] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.958115 10070 consensus_queue.cc:260] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.959386 10046 heartbeater.cc:502] Master 127.0.104.62:40227 was elected leader, sending a full tablet report...
I20250110 20:54:11.959717 10071 tablet_bootstrap.cc:492] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:11.959861 10071 ts_tablet_manager.cc:1397] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.008s
I20250110 20:54:11.960287 10070 ts_tablet_manager.cc:1428] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20250110 20:54:11.960656 10070 tablet_bootstrap.cc:492] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:11.961045 10071 raft_consensus.cc:357] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.961257 10071 raft_consensus.cc:738] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.961442 10071 consensus_queue.cc:260] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.962030 10071 ts_tablet_manager.cc:1428] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 20:54:11.962239 10070 tablet_bootstrap.cc:654] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.962289 10071 tablet_bootstrap.cc:492] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:11.963799 10071 tablet_bootstrap.cc:654] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.963821 10069 raft_consensus.cc:357] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:11.964243 10069 raft_consensus.cc:738] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.964710 10069 consensus_queue.cc:260] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:11.983922 10071 tablet_bootstrap.cc:492] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:11.984122 10071 ts_tablet_manager.cc:1397] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.011s
I20250110 20:54:11.984508 9916 heartbeater.cc:502] Master 127.0.104.62:40227 was elected leader, sending a full tablet report...
I20250110 20:54:11.984604 10069 ts_tablet_manager.cc:1428] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.029s user 0.005s sys 0.023s
I20250110 20:54:11.984803 10071 raft_consensus.cc:357] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.984954 10069 tablet_bootstrap.cc:492] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:11.985016 10071 raft_consensus.cc:738] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.985265 10071 consensus_queue.cc:260] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.985831 10071 ts_tablet_manager.cc:1428] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 20:54:11.986272 10071 tablet_bootstrap.cc:492] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:11.986655 10070 tablet_bootstrap.cc:492] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:11.986687 10069 tablet_bootstrap.cc:654] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.986835 10070 ts_tablet_manager.cc:1397] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.019s
I20250110 20:54:11.987679 10070 raft_consensus.cc:357] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.987886 10070 raft_consensus.cc:738] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.988016 10071 tablet_bootstrap.cc:654] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.988077 10070 consensus_queue.cc:260] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.988545 10070 ts_tablet_manager.cc:1428] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250110 20:54:11.991369 10070 tablet_bootstrap.cc:492] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:11.993054 10070 tablet_bootstrap.cc:654] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:11.996116 10071 tablet_bootstrap.cc:492] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:11.996259 10071 ts_tablet_manager.cc:1397] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250110 20:54:11.996325 10070 tablet_bootstrap.cc:492] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:11.996460 10070 ts_tablet_manager.cc:1397] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250110 20:54:11.996497 10069 tablet_bootstrap.cc:492] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:11.996636 10069 ts_tablet_manager.cc:1397] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250110 20:54:11.996891 10071 raft_consensus.cc:357] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.997089 10071 raft_consensus.cc:738] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.997076 10070 raft_consensus.cc:357] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.997287 10070 raft_consensus.cc:738] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.997262 10071 consensus_queue.cc:260] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.997475 10069 raft_consensus.cc:357] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.997666 10069 raft_consensus.cc:738] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:11.997761 10071 ts_tablet_manager.cc:1428] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:11.997830 10069 consensus_queue.cc:260] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.998021 10071 tablet_bootstrap.cc:492] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:11.998407 10069 ts_tablet_manager.cc:1428] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 20:54:11.998526 10070 consensus_queue.cc:260] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:11.999049 10070 ts_tablet_manager.cc:1428] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 20:54:11.999337 10070 tablet_bootstrap.cc:492] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
W20250110 20:54:11.999392 10047 tablet.cc:2367] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:54:11.999742 10071 tablet_bootstrap.cc:654] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.001071 10070 tablet_bootstrap.cc:654] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.003360 10069 tablet_bootstrap.cc:492] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.005673 10069 tablet_bootstrap.cc:654] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.007999 10071 tablet_bootstrap.cc:492] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:12.008131 10071 ts_tablet_manager.cc:1397] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20250110 20:54:12.008399 10070 tablet_bootstrap.cc:492] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.008527 10070 ts_tablet_manager.cc:1397] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20250110 20:54:12.008679 10071 raft_consensus.cc:357] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.008834 10071 raft_consensus.cc:738] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.009008 10071 consensus_queue.cc:260] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.009065 10070 raft_consensus.cc:357] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.009224 10070 raft_consensus.cc:738] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.009405 10071 ts_tablet_manager.cc:1428] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:12.009375 10070 consensus_queue.cc:260] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.009598 10071 tablet_bootstrap.cc:492] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:12.009876 10070 ts_tablet_manager.cc:1428] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250110 20:54:12.010135 10070 tablet_bootstrap.cc:492] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:12.011127 10071 tablet_bootstrap.cc:654] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.011659 10070 tablet_bootstrap.cc:654] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.013960 10069 tablet_bootstrap.cc:492] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.016410 10069 ts_tablet_manager.cc:1397] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250110 20:54:12.016741 10070 tablet_bootstrap.cc:492] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.016872 10071 tablet_bootstrap.cc:492] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:12.016880 10070 ts_tablet_manager.cc:1397] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20250110 20:54:12.017002 10071 ts_tablet_manager.cc:1397] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20250110 20:54:12.017154 10069 raft_consensus.cc:357] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.017362 10069 raft_consensus.cc:738] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.017372 10070 raft_consensus.cc:357] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.017499 10070 raft_consensus.cc:738] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.017530 10069 consensus_queue.cc:260] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.017629 10070 consensus_queue.cc:260] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.017637 10071 raft_consensus.cc:357] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.017864 10071 raft_consensus.cc:738] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.017998 10069 ts_tablet_manager.cc:1428] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:12.018023 10070 ts_tablet_manager.cc:1428] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:12.018028 10071 consensus_queue.cc:260] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.018239 10069 tablet_bootstrap.cc:492] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.018244 10070 tablet_bootstrap.cc:492] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:12.018458 10071 ts_tablet_manager.cc:1428] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:12.018680 10071 tablet_bootstrap.cc:492] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:12.019744 10069 tablet_bootstrap.cc:654] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.019804 10070 tablet_bootstrap.cc:654] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.020136 10071 tablet_bootstrap.cc:654] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.032212 10070 tablet_bootstrap.cc:492] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.032366 10070 ts_tablet_manager.cc:1397] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.011s
I20250110 20:54:12.032788 10071 tablet_bootstrap.cc:492] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:12.032953 10071 ts_tablet_manager.cc:1397] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.011s
I20250110 20:54:12.033077 10070 raft_consensus.cc:357] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.033229 10069 tablet_bootstrap.cc:492] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.033329 10070 raft_consensus.cc:738] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.033390 10069 ts_tablet_manager.cc:1397] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20250110 20:54:12.033517 10070 consensus_queue.cc:260] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.033672 10071 raft_consensus.cc:357] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.033881 10071 raft_consensus.cc:738] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.034180 10069 raft_consensus.cc:357] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.034152 10071 consensus_queue.cc:260] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.034595 10070 ts_tablet_manager.cc:1428] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 20:54:12.034909 10070 tablet_bootstrap.cc:492] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:12.036535 10070 tablet_bootstrap.cc:654] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.036669 10071 ts_tablet_manager.cc:1428] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250110 20:54:12.036803 10069 raft_consensus.cc:738] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.037413 10069 consensus_queue.cc:260] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.038040 10071 tablet_bootstrap.cc:492] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:12.039157 10069 ts_tablet_manager.cc:1428] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250110 20:54:12.039552 10069 tablet_bootstrap.cc:492] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.039587 10071 tablet_bootstrap.cc:654] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.042511 10069 tablet_bootstrap.cc:654] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.042634 10070 tablet_bootstrap.cc:492] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.042858 10070 ts_tablet_manager.cc:1397] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20250110 20:54:12.043558 10070 raft_consensus.cc:357] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.043779 10070 raft_consensus.cc:738] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.043967 10070 consensus_queue.cc:260] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.044303 10069 tablet_bootstrap.cc:492] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.044407 10070 ts_tablet_manager.cc:1428] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250110 20:54:12.044438 10069 ts_tablet_manager.cc:1397] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250110 20:54:12.044696 10070 tablet_bootstrap.cc:492] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:12.045164 10069 raft_consensus.cc:357] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.045363 10069 raft_consensus.cc:738] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.045574 10069 consensus_queue.cc:260] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.046160 10069 ts_tablet_manager.cc:1428] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 20:54:12.046175 10070 tablet_bootstrap.cc:654] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.046445 10069 tablet_bootstrap.cc:492] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.047977 10069 tablet_bootstrap.cc:654] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.049444 10071 tablet_bootstrap.cc:492] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:12.049582 10071 ts_tablet_manager.cc:1397] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20250110 20:54:12.050173 10071 raft_consensus.cc:357] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.050388 10071 raft_consensus.cc:738] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.050576 10071 consensus_queue.cc:260] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.050988 10071 ts_tablet_manager.cc:1428] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 20:54:12.051256 10071 tablet_bootstrap.cc:492] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:12.052691 10071 tablet_bootstrap.cc:654] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.055749 10069 tablet_bootstrap.cc:492] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.055881 10069 ts_tablet_manager.cc:1397] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20250110 20:54:12.055954 10070 tablet_bootstrap.cc:492] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.055948 10071 tablet_bootstrap.cc:492] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:12.056094 10071 ts_tablet_manager.cc:1397] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20250110 20:54:12.056095 10070 ts_tablet_manager.cc:1397] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.011s user 0.011s sys 0.000s
I20250110 20:54:12.056421 10069 raft_consensus.cc:357] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.056612 10069 raft_consensus.cc:738] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.056696 10071 raft_consensus.cc:357] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.056766 10069 consensus_queue.cc:260] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.056721 10070 raft_consensus.cc:357] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:12.056915 10070 raft_consensus.cc:738] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.057192 10069 ts_tablet_manager.cc:1428] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:12.057233 10070 consensus_queue.cc:260] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:12.057466 10069 tablet_bootstrap.cc:492] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.057710 10071 raft_consensus.cc:738] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.057811 10070 ts_tablet_manager.cc:1428] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250110 20:54:12.057938 10071 consensus_queue.cc:260] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.058069 10070 tablet_bootstrap.cc:492] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:12.058414 10071 ts_tablet_manager.cc:1428] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250110 20:54:12.058696 10071 tablet_bootstrap.cc:492] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16: Bootstrap starting.
I20250110 20:54:12.059002 10069 tablet_bootstrap.cc:654] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.059617 10070 tablet_bootstrap.cc:654] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.060282 10071 tablet_bootstrap.cc:654] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.067760 10070 tablet_bootstrap.cc:492] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.067893 10070 ts_tablet_manager.cc:1397] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20250110 20:54:12.067979 10069 tablet_bootstrap.cc:492] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.068118 10069 ts_tablet_manager.cc:1397] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20250110 20:54:12.068463 10071 tablet_bootstrap.cc:492] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16: No bootstrap required, opened a new log
I20250110 20:54:12.068615 10071 ts_tablet_manager.cc:1397] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20250110 20:54:12.068516 10070 raft_consensus.cc:357] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.068710 10070 raft_consensus.cc:738] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.068667 10069 raft_consensus.cc:357] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.068840 10069 raft_consensus.cc:738] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.068886 10070 consensus_queue.cc:260] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.069288 10071 raft_consensus.cc:357] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:12.069293 10069 consensus_queue.cc:260] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.069782 10069 ts_tablet_manager.cc:1428] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250110 20:54:12.070079 10069 tablet_bootstrap.cc:492] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.071362 10070 ts_tablet_manager.cc:1428] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250110 20:54:12.071722 10069 tablet_bootstrap.cc:654] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.072176 10070 tablet_bootstrap.cc:492] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6: Bootstrap starting.
I20250110 20:54:12.072770 10071 raft_consensus.cc:738] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e59427cf93f4bd480e35b4ba3207e16, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.073035 10071 consensus_queue.cc:260] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:12.073696 10070 tablet_bootstrap.cc:654] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.075124 10071 ts_tablet_manager.cc:1428] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250110 20:54:12.076261 10069 tablet_bootstrap.cc:492] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.076705 10069 ts_tablet_manager.cc:1397] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20250110 20:54:12.076915 10070 tablet_bootstrap.cc:492] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6: No bootstrap required, opened a new log
I20250110 20:54:12.077054 10070 ts_tablet_manager.cc:1397] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20250110 20:54:12.077338 10069 raft_consensus.cc:357] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.077492 10069 raft_consensus.cc:738] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.077668 10069 consensus_queue.cc:260] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.077637 10070 raft_consensus.cc:357] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.077821 10070 raft_consensus.cc:738] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c6369336864e6a88a75d229d4448b6, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.078042 10069 ts_tablet_manager.cc:1428] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250110 20:54:12.077991 10070 consensus_queue.cc:260] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.078352 10069 tablet_bootstrap.cc:492] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.078576 10070 ts_tablet_manager.cc:1428] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250110 20:54:12.079820 10069 tablet_bootstrap.cc:654] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.081472 10069 tablet_bootstrap.cc:492] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.081629 10069 ts_tablet_manager.cc:1397] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 20:54:12.082238 10069 raft_consensus.cc:357] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.082393 10069 raft_consensus.cc:738] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.082510 10069 consensus_queue.cc:260] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.082902 10069 ts_tablet_manager.cc:1428] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250110 20:54:12.083143 10069 tablet_bootstrap.cc:492] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Bootstrap starting.
I20250110 20:54:12.084558 10069 tablet_bootstrap.cc:654] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Neither blocks nor log segments found. Creating new log.
I20250110 20:54:12.086129 10069 tablet_bootstrap.cc:492] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: No bootstrap required, opened a new log
I20250110 20:54:12.086265 10069 ts_tablet_manager.cc:1397] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250110 20:54:12.086784 10069 raft_consensus.cc:357] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.086942 10069 raft_consensus.cc:738] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Initialized, Role: FOLLOWER
I20250110 20:54:12.087088 10069 consensus_queue.cc:260] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.087479 10069 ts_tablet_manager.cc:1428] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
W20250110 20:54:12.097071 9917 tablet.cc:2367] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:54:12.121721 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4b110d55d2e34081844cb4797d97e289"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.121945 9871 raft_consensus.cc:491] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.122079 9871 raft_consensus.cc:3054] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.123976 9871 raft_consensus.cc:513] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:12.125259 9871 leader_election.cc:290] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289), 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893)
I20250110 20:54:12.132089 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b110d55d2e34081844cb4797d97e289" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.132500 9741 raft_consensus.cc:3054] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.132614 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b110d55d2e34081844cb4797d97e289" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.132956 10001 raft_consensus.cc:3054] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.134505 9741 raft_consensus.cc:2463] T 4b110d55d2e34081844cb4797d97e289 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.134948 10001 raft_consensus.cc:2463] T 4b110d55d2e34081844cb4797d97e289 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.135047 9805 leader_election.cc:304] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.135433 10080 raft_consensus.cc:2798] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.135848 10080 raft_consensus.cc:695] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.136303 10080 consensus_queue.cc:237] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } }
I20250110 20:54:12.140331 9600 catalog_manager.cc:5526] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.148168 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8ddff8ac3c8a433a8587a9a657000089"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.148326 9871 raft_consensus.cc:491] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.148396 9871 raft_consensus.cc:3054] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.150043 9871 raft_consensus.cc:513] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.150494 9871 leader_election.cc:290] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.150847 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddff8ac3c8a433a8587a9a657000089" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.150949 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ddff8ac3c8a433a8587a9a657000089" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.151055 9741 raft_consensus.cc:3054] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.151119 10001 raft_consensus.cc:3054] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.153621 9741 raft_consensus.cc:2463] T 8ddff8ac3c8a433a8587a9a657000089 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.153621 10001 raft_consensus.cc:2463] T 8ddff8ac3c8a433a8587a9a657000089 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.153985 9805 leader_election.cc:304] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 23c6369336864e6a88a75d229d4448b6, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.154247 10080 raft_consensus.cc:2798] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.154392 10080 raft_consensus.cc:695] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.154606 10080 consensus_queue.cc:237] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.156929 9600 catalog_manager.cc:5526] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.164798 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "56acdd81451e447ebb4bf1943947f52c"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.164961 9871 raft_consensus.cc:491] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.165072 9871 raft_consensus.cc:3054] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.166906 9871 raft_consensus.cc:513] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.167397 9871 leader_election.cc:290] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.167863 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56acdd81451e447ebb4bf1943947f52c" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.167871 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56acdd81451e447ebb4bf1943947f52c" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.168071 10001 raft_consensus.cc:3054] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.168071 9741 raft_consensus.cc:3054] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.170693 10001 raft_consensus.cc:2463] T 56acdd81451e447ebb4bf1943947f52c P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.170693 9741 raft_consensus.cc:2463] T 56acdd81451e447ebb4bf1943947f52c P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.171109 9807 leader_election.cc:304] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 23c6369336864e6a88a75d229d4448b6; no voters:
I20250110 20:54:12.171386 10080 raft_consensus.cc:2798] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.171523 10080 raft_consensus.cc:695] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.171768 10080 consensus_queue.cc:237] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.174295 9600 catalog_manager.cc:5526] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.182247 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a50e6ed61c574319866dbcadd6c81632"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.182407 9871 raft_consensus.cc:491] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.182480 9871 raft_consensus.cc:3054] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.184178 9871 raft_consensus.cc:513] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.184665 9871 leader_election.cc:290] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.185024 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50e6ed61c574319866dbcadd6c81632" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.185158 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a50e6ed61c574319866dbcadd6c81632" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.185235 9741 raft_consensus.cc:3054] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.185381 10001 raft_consensus.cc:3054] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.187134 9741 raft_consensus.cc:2463] T a50e6ed61c574319866dbcadd6c81632 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.187270 10001 raft_consensus.cc:2463] T a50e6ed61c574319866dbcadd6c81632 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.187505 9805 leader_election.cc:304] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.187758 10080 raft_consensus.cc:2798] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.187903 10080 raft_consensus.cc:695] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.188212 10080 consensus_queue.cc:237] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.190675 9600 catalog_manager.cc:5526] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
W20250110 20:54:12.194597 9787 tablet.cc:2367] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:54:12.199579 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "105ab07a05d8485297bc4997e40875d6"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.199734 9871 raft_consensus.cc:491] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.199831 9871 raft_consensus.cc:3054] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.201488 9871 raft_consensus.cc:513] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.201969 9871 leader_election.cc:290] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.202308 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "105ab07a05d8485297bc4997e40875d6" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.202400 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "105ab07a05d8485297bc4997e40875d6" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.202487 9741 raft_consensus.cc:3054] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.202579 10001 raft_consensus.cc:3054] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.204459 9741 raft_consensus.cc:2463] T 105ab07a05d8485297bc4997e40875d6 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.204563 10001 raft_consensus.cc:2463] T 105ab07a05d8485297bc4997e40875d6 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.204783 9805 leader_election.cc:304] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.205034 10080 raft_consensus.cc:2798] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.205173 10080 raft_consensus.cc:695] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.205453 10080 consensus_queue.cc:237] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.207631 9600 catalog_manager.cc:5526] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.216313 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d0d07a127584348bbbaf0706ab35c97"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.216468 9871 raft_consensus.cc:491] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.216536 9871 raft_consensus.cc:3054] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.218137 9871 raft_consensus.cc:513] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.218575 9871 leader_election.cc:290] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.218986 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0d07a127584348bbbaf0706ab35c97" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.219005 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d0d07a127584348bbbaf0706ab35c97" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.219175 10001 raft_consensus.cc:3054] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.219189 9741 raft_consensus.cc:3054] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.220963 9741 raft_consensus.cc:2463] T 6d0d07a127584348bbbaf0706ab35c97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.221004 10001 raft_consensus.cc:2463] T 6d0d07a127584348bbbaf0706ab35c97 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.221304 9805 leader_election.cc:304] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.221609 10080 raft_consensus.cc:2798] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.221915 10080 raft_consensus.cc:695] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.222147 10080 consensus_queue.cc:237] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.224313 9600 catalog_manager.cc:5526] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.232875 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77ec0f41406a451e94a103173bed7483"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.233021 9871 raft_consensus.cc:491] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.233117 9871 raft_consensus.cc:3054] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.234858 9871 raft_consensus.cc:513] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.235379 9871 leader_election.cc:290] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.235695 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ec0f41406a451e94a103173bed7483" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.235787 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ec0f41406a451e94a103173bed7483" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.235872 9741 raft_consensus.cc:3054] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.235970 10001 raft_consensus.cc:3054] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.238415 9741 raft_consensus.cc:2463] T 77ec0f41406a451e94a103173bed7483 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.238452 10001 raft_consensus.cc:2463] T 77ec0f41406a451e94a103173bed7483 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.238799 9807 leader_election.cc:304] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 23c6369336864e6a88a75d229d4448b6; no voters:
I20250110 20:54:12.239053 10080 raft_consensus.cc:2798] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.239202 10080 raft_consensus.cc:695] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.239423 10080 consensus_queue.cc:237] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.241616 9600 catalog_manager.cc:5526] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.249284 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2e7e7cd1b41c4d2b962b7dad46f5468e"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.249440 9871 raft_consensus.cc:491] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.249521 9871 raft_consensus.cc:3054] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.251363 9871 raft_consensus.cc:513] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.251794 9871 leader_election.cc:290] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.252177 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e7e7cd1b41c4d2b962b7dad46f5468e" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.252216 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e7e7cd1b41c4d2b962b7dad46f5468e" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.252367 10001 raft_consensus.cc:3054] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.252408 9741 raft_consensus.cc:3054] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.253990 10001 raft_consensus.cc:2463] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.253994 9741 raft_consensus.cc:2463] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.254312 9805 leader_election.cc:304] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.254542 10080 raft_consensus.cc:2798] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.254703 10080 raft_consensus.cc:695] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.254921 10080 consensus_queue.cc:237] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.256986 9600 catalog_manager.cc:5526] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.264881 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2f3ba987d1ed429d9ddf3d7e98d90d97"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.265022 9871 raft_consensus.cc:491] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.265118 9871 raft_consensus.cc:3054] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.266708 9871 raft_consensus.cc:513] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.267237 9871 leader_election.cc:290] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.267571 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ba987d1ed429d9ddf3d7e98d90d97" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.267750 9741 raft_consensus.cc:3054] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.267844 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f3ba987d1ed429d9ddf3d7e98d90d97" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.268023 10001 raft_consensus.cc:3054] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.269577 9741 raft_consensus.cc:2463] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.269789 10001 raft_consensus.cc:2463] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.269881 9805 leader_election.cc:304] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 7e59427cf93f4bd480e35b4ba3207e16; no voters:
I20250110 20:54:12.270179 10080 raft_consensus.cc:2798] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.270296 10080 raft_consensus.cc:695] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.270496 10080 consensus_queue.cc:237] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.272748 9600 catalog_manager.cc:5526] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.280445 9871 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0fb8ed70417453aa19acb9c5125a62e"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.280588 9871 raft_consensus.cc:491] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250110 20:54:12.280658 9871 raft_consensus.cc:3054] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.282308 9871 raft_consensus.cc:513] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.282725 9871 leader_election.cc:290] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Requested vote from peers 7e59427cf93f4bd480e35b4ba3207e16 (127.0.104.1:41893), 23c6369336864e6a88a75d229d4448b6 (127.0.104.3:35289)
I20250110 20:54:12.283111 9741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb8ed70417453aa19acb9c5125a62e" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e59427cf93f4bd480e35b4ba3207e16"
I20250110 20:54:12.283161 10001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0fb8ed70417453aa19acb9c5125a62e" candidate_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "23c6369336864e6a88a75d229d4448b6"
I20250110 20:54:12.283313 9741 raft_consensus.cc:3054] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.283342 10001 raft_consensus.cc:3054] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:54:12.284916 10001 raft_consensus.cc:2463] T f0fb8ed70417453aa19acb9c5125a62e P 23c6369336864e6a88a75d229d4448b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.284957 9741 raft_consensus.cc:2463] T f0fb8ed70417453aa19acb9c5125a62e P 7e59427cf93f4bd480e35b4ba3207e16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ece0b1ce9554085b5daaafa501fd4fc in term 1.
I20250110 20:54:12.285255 9807 leader_election.cc:304] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ece0b1ce9554085b5daaafa501fd4fc, 23c6369336864e6a88a75d229d4448b6; no voters:
I20250110 20:54:12.285528 10080 raft_consensus.cc:2798] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:54:12.285687 10080 raft_consensus.cc:695] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Leader. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.285897 10080 consensus_queue.cc:237] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.288044 9600 catalog_manager.cc:5526] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ece0b1ce9554085b5daaafa501fd4fc (127.0.104.2). New cstate: current_term: 1 leader_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } health_report { overall_health: UNKNOWN } } }
I20250110 20:54:12.296012 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b110d55d2e34081844cb4797d97e289 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.296643 10120 tablet_replica.cc:331] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.296998 10120 raft_consensus.cc:2238] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:12.297400 10120 pending_rounds.cc:62] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.297508 10120 pending_rounds.cc:69] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877883954364416 op_type: NO_OP noop_request { }
I20250110 20:54:12.297643 10120 raft_consensus.cc:2883] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.297732 10120 raft_consensus.cc:2267] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.299052 10120 ts_tablet_manager.cc:1905] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.303093 10120 ts_tablet_manager.cc:1918] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.303222 10120 log.cc:1198] T 4b110d55d2e34081844cb4797d97e289 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/4b110d55d2e34081844cb4797d97e289
I20250110 20:54:12.311390 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ddff8ac3c8a433a8587a9a657000089 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.311579 10120 tablet_replica.cc:331] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.311717 10120 raft_consensus.cc:2238] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:12.311868 10120 pending_rounds.cc:62] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.311925 10120 pending_rounds.cc:69] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884026957824 op_type: NO_OP noop_request { }
I20250110 20:54:12.312016 10120 raft_consensus.cc:2883] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.312101 10120 raft_consensus.cc:2267] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.312700 10120 ts_tablet_manager.cc:1905] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.316368 10120 ts_tablet_manager.cc:1918] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.316452 10120 log.cc:1198] T 8ddff8ac3c8a433a8587a9a657000089 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/8ddff8ac3c8a433a8587a9a657000089
I20250110 20:54:12.324373 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 56acdd81451e447ebb4bf1943947f52c with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.324568 10120 tablet_replica.cc:331] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.324693 10120 raft_consensus.cc:2238] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:12.324853 10120 pending_rounds.cc:62] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.324896 10120 pending_rounds.cc:69] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884097560576 op_type: NO_OP noop_request { }
I20250110 20:54:12.324991 10120 raft_consensus.cc:2883] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.325058 10120 raft_consensus.cc:2267] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.325668 10120 ts_tablet_manager.cc:1905] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.329092 10120 ts_tablet_manager.cc:1918] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.329188 10120 log.cc:1198] T 56acdd81451e447ebb4bf1943947f52c P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/56acdd81451e447ebb4bf1943947f52c
I20250110 20:54:12.337049 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet a50e6ed61c574319866dbcadd6c81632 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.337232 10120 tablet_replica.cc:331] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.337363 10120 raft_consensus.cc:2238] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:12.337512 10120 pending_rounds.cc:62] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.337566 10120 pending_rounds.cc:69] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884164571136 op_type: NO_OP noop_request { }
I20250110 20:54:12.337638 10120 raft_consensus.cc:2883] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.337714 10120 raft_consensus.cc:2267] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.338326 10120 ts_tablet_manager.cc:1905] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.341897 10120 ts_tablet_manager.cc:1918] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.341996 10120 log.cc:1198] T a50e6ed61c574319866dbcadd6c81632 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/a50e6ed61c574319866dbcadd6c81632
I20250110 20:54:12.354754 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 105ab07a05d8485297bc4997e40875d6 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.354980 10120 tablet_replica.cc:331] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.355129 10120 raft_consensus.cc:2238] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:54:12.355342 10120 pending_rounds.cc:62] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.355394 10120 pending_rounds.cc:69] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884235227136 op_type: NO_OP noop_request { }
I20250110 20:54:12.355530 10120 raft_consensus.cc:2883] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.355628 10120 raft_consensus.cc:2267] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.356353 10120 ts_tablet_manager.cc:1905] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.360205 10120 ts_tablet_manager.cc:1918] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.360316 10120 log.cc:1198] T 105ab07a05d8485297bc4997e40875d6 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/105ab07a05d8485297bc4997e40875d6
I20250110 20:54:12.374186 9871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d0d07a127584348bbbaf0706ab35c97"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.374375 9871 raft_consensus.cc:589] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Received request to step down
I20250110 20:54:12.374467 9871 raft_consensus.cc:3049] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:12.374534 9871 raft_consensus.cc:738] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.374838 9871 consensus_queue.cc:260] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.375098 9871 raft_consensus.cc:3054] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:12.388523 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d0d07a127584348bbbaf0706ab35c97 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.388734 10120 tablet_replica.cc:331] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.388896 10120 raft_consensus.cc:2238] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:12.389027 10120 pending_rounds.cc:62] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.389093 10120 pending_rounds.cc:69] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884303634432 op_type: NO_OP noop_request { }
I20250110 20:54:12.389211 10120 raft_consensus.cc:2883] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.389325 10120 raft_consensus.cc:2267] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.390044 10120 ts_tablet_manager.cc:1905] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.394058 10120 ts_tablet_manager.cc:1918] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.394148 10120 log.cc:1198] T 6d0d07a127584348bbbaf0706ab35c97 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/6d0d07a127584348bbbaf0706ab35c97
I20250110 20:54:12.402169 9871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77ec0f41406a451e94a103173bed7483"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.402310 9871 raft_consensus.cc:589] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Received request to step down
I20250110 20:54:12.402381 9871 raft_consensus.cc:3049] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:12.402427 9871 raft_consensus.cc:738] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.402601 9871 consensus_queue.cc:260] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.402815 9871 raft_consensus.cc:3054] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:12.410563 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 77ec0f41406a451e94a103173bed7483 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.410802 10120 tablet_replica.cc:331] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.410969 10120 raft_consensus.cc:2238] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:12.411096 10120 pending_rounds.cc:62] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.411139 10120 pending_rounds.cc:69] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884374151168 op_type: NO_OP noop_request { }
I20250110 20:54:12.411228 10120 raft_consensus.cc:2883] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.411293 10120 raft_consensus.cc:2267] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.411926 10120 ts_tablet_manager.cc:1905] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.415563 10120 ts_tablet_manager.cc:1918] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.415688 10120 log.cc:1198] T 77ec0f41406a451e94a103173bed7483 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/77ec0f41406a451e94a103173bed7483
I20250110 20:54:12.423600 9871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2e7e7cd1b41c4d2b962b7dad46f5468e"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.423733 9871 raft_consensus.cc:589] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Received request to step down
I20250110 20:54:12.423803 9871 raft_consensus.cc:3049] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:12.423849 9871 raft_consensus.cc:738] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.424022 9871 consensus_queue.cc:260] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.424199 9871 raft_consensus.cc:3054] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:12.431890 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e7e7cd1b41c4d2b962b7dad46f5468e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.432081 10120 tablet_replica.cc:331] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.432214 10120 raft_consensus.cc:2238] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:12.432314 10120 pending_rounds.cc:62] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.432365 10120 pending_rounds.cc:69] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884437704704 op_type: NO_OP noop_request { }
I20250110 20:54:12.432444 10120 raft_consensus.cc:2883] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.432519 10120 raft_consensus.cc:2267] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.433140 10120 ts_tablet_manager.cc:1905] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.436677 10120 ts_tablet_manager.cc:1918] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.436761 10120 log.cc:1198] T 2e7e7cd1b41c4d2b962b7dad46f5468e P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/2e7e7cd1b41c4d2b962b7dad46f5468e
I20250110 20:54:12.444819 9871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2f3ba987d1ed429d9ddf3d7e98d90d97"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.444985 9871 raft_consensus.cc:589] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Received request to step down
I20250110 20:54:12.445086 9871 raft_consensus.cc:3049] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:12.445142 9871 raft_consensus.cc:738] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.445345 9871 consensus_queue.cc:260] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.445583 9871 raft_consensus.cc:3054] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:12.458796 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f3ba987d1ed429d9ddf3d7e98d90d97 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.459034 10120 tablet_replica.cc:331] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.459188 10120 raft_consensus.cc:2238] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:12.459332 10120 pending_rounds.cc:62] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.459383 10120 pending_rounds.cc:69] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884501712896 op_type: NO_OP noop_request { }
I20250110 20:54:12.459506 10120 raft_consensus.cc:2883] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.459594 10120 raft_consensus.cc:2267] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.460268 10120 ts_tablet_manager.cc:1905] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.464053 10120 ts_tablet_manager.cc:1918] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.464164 10120 log.cc:1198] T 2f3ba987d1ed429d9ddf3d7e98d90d97 P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f3ba987d1ed429d9ddf3d7e98d90d97
I20250110 20:54:12.472404 9871 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0fb8ed70417453aa19acb9c5125a62e"
dest_uuid: "0ece0b1ce9554085b5daaafa501fd4fc"
from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.472540 9871 raft_consensus.cc:589] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Received request to step down
I20250110 20:54:12.472625 9871 raft_consensus.cc:3049] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:54:12.472682 9871 raft_consensus.cc:738] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0ece0b1ce9554085b5daaafa501fd4fc, State: Running, Role: LEADER
I20250110 20:54:12.472874 9871 consensus_queue.cc:260] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e59427cf93f4bd480e35b4ba3207e16" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 41893 } } peers { permanent_uuid: "0ece0b1ce9554085b5daaafa501fd4fc" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35491 } } peers { permanent_uuid: "23c6369336864e6a88a75d229d4448b6" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35289 } }
I20250110 20:54:12.473084 9871 raft_consensus.cc:3054] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:54:12.481238 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet f0fb8ed70417453aa19acb9c5125a62e with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:52082
I20250110 20:54:12.481451 10120 tablet_replica.cc:331] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: stopping tablet replica
I20250110 20:54:12.481608 10120 raft_consensus.cc:2238] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:54:12.481729 10120 pending_rounds.cc:62] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Trying to abort 1 pending ops.
I20250110 20:54:12.481796 10120 pending_rounds.cc:69] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112877884564492288 op_type: NO_OP noop_request { }
I20250110 20:54:12.481894 10120 raft_consensus.cc:2883] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:54:12.482002 10120 raft_consensus.cc:2267] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:54:12.482681 10120 ts_tablet_manager.cc:1905] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250110 20:54:12.486331 10120 ts_tablet_manager.cc:1918] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250110 20:54:12.486426 10120 log.cc:1198] T f0fb8ed70417453aa19acb9c5125a62e P 0ece0b1ce9554085b5daaafa501fd4fc: Deleting WAL directory at /tmp/dist-test-taskrkqplS/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736542259391478-416-0/raft_consensus-itest-cluster/ts-1/wal/wals/f0fb8ed70417453aa19acb9c5125a62e
I20250110 20:54:12.495349 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9659
I20250110 20:54:12.500597 9781 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:12.594488 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9659
I20250110 20:54:12.609053 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9789
I20250110 20:54:12.614521 9911 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:12.723932 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9789
I20250110 20:54:12.739171 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9919
I20250110 20:54:12.744279 10041 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:12.835717 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9919
I20250110 20:54:12.848848 416 external_mini_cluster.cc:1661] Attempting to check leaks for /tmp/dist-test-taskrkqplS/build/asan/bin/kudu pid 9569
I20250110 20:54:12.849774 9630 generic_service.cc:196] Checking for leaks (request via RPC)
I20250110 20:54:12.951586 416 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskrkqplS/build/asan/bin/kudu with pid 9569
2025-01-10T20:54:12Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (2959 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (4645 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (193528 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20250110 20:54:12.990267 416 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 2 messages since previous log ~15 seconds ago