[==========] Running 14 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 12 tests from RaftConsensusElectionITest
[ RUN ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250114 03:02:52.381975 1461 test_util.cc:274] Using random seed: -872578300
I20250114 03:02:52.388788 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:02:52.388850 1461 ts_itest-base.cc:113] --------------
I20250114 03:02:52.388862 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:02:52.388877 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:02:52.388903 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:02:52Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:02:52Z Disabled control of system clock
I20250114 03:02:52.398818 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:37099
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:37099
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250114 03:02:52.504588 1473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:02:52.504805 1473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:02:52.504885 1473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:02:52.507985 1473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:02:52.508037 1473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:02:52.508056 1473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:02:52.508078 1473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:02:52.511693 1473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:37099
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.1473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:02:52.512766 1473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:02:52.513968 1473 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:02:52.521200 1478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:52.522228 1481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:02:52.522958 1473 server_base.cc:1034] running on GCE node
W20250114 03:02:52.523258 1479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:02:52.523730 1473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:02:52.524631 1473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:02:52.525789 1473 hybrid_clock.cc:648] HybridClock initialized: now 1736823772525776 us; error 25 us; skew 500 ppm
I20250114 03:02:52.527710 1473 webserver.cc:458] Webserver started at http://127.1.109.126:34873/ using document root <none> and password file <none>
I20250114 03:02:52.528311 1473 fs_manager.cc:362] Metadata directory not provided
I20250114 03:02:52.528370 1473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:02:52.528540 1473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:02:52.530228 1473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data/instance:
uuid: "a5e42d3f5b224492a308f67b4f8006fb"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.530692 1473 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal/instance:
uuid: "a5e42d3f5b224492a308f67b4f8006fb"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.534103 1473 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:02:52.536240 1487 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.537285 1473 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:02:52.537412 1473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal
uuid: "a5e42d3f5b224492a308f67b4f8006fb"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.537496 1473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:02:52.548943 1473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:02:52.549513 1473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:02:52.549660 1473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:02:52.556285 1473 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:37099
I20250114 03:02:52.556300 1539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:37099 every 8 connection(s)
I20250114 03:02:52.557265 1473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/data/info.pb
I20250114 03:02:52.559803 1540 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:02:52.564249 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 1473
I20250114 03:02:52.564404 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/master-0/wal/instance
I20250114 03:02:52.565505 1540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb: Bootstrap starting.
I20250114 03:02:52.568308 1540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb: Neither blocks nor log segments found. Creating new log.
I20250114 03:02:52.569182 1540 log.cc:826] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb: Log is configured to *not* fsync() on all Append() calls
I20250114 03:02:52.570950 1540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb: No bootstrap required, opened a new log
I20250114 03:02:52.574467 1540 raft_consensus.cc:357] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e42d3f5b224492a308f67b4f8006fb" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 37099 } }
I20250114 03:02:52.574700 1540 raft_consensus.cc:383] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:02:52.574756 1540 raft_consensus.cc:738] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e42d3f5b224492a308f67b4f8006fb, State: Initialized, Role: FOLLOWER
I20250114 03:02:52.575205 1540 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e42d3f5b224492a308f67b4f8006fb" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 37099 } }
I20250114 03:02:52.575335 1540 raft_consensus.cc:397] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:02:52.575387 1540 raft_consensus.cc:491] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:02:52.575500 1540 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:02:52.576256 1540 raft_consensus.cc:513] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e42d3f5b224492a308f67b4f8006fb" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 37099 } }
I20250114 03:02:52.576628 1540 leader_election.cc:304] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5e42d3f5b224492a308f67b4f8006fb; no voters:
I20250114 03:02:52.577020 1540 leader_election.cc:290] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:02:52.577113 1545 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:02:52.577328 1545 raft_consensus.cc:695] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [term 1 LEADER]: Becoming Leader. State: Replica: a5e42d3f5b224492a308f67b4f8006fb, State: Running, Role: LEADER
I20250114 03:02:52.577667 1545 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e42d3f5b224492a308f67b4f8006fb" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 37099 } }
I20250114 03:02:52.578249 1540 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:02:52.579277 1547 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5e42d3f5b224492a308f67b4f8006fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e42d3f5b224492a308f67b4f8006fb" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 37099 } } }
I20250114 03:02:52.579407 1547 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [sys.catalog]: This master's current role is: LEADER
I20250114 03:02:52.579769 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5e42d3f5b224492a308f67b4f8006fb. Latest consensus state: current_term: 1 leader_uuid: "a5e42d3f5b224492a308f67b4f8006fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5e42d3f5b224492a308f67b4f8006fb" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 37099 } } }
I20250114 03:02:52.579908 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb [sys.catalog]: This master's current role is: LEADER
I20250114 03:02:52.580775 1554 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:02:52.584074 1554 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:02:52.589154 1554 catalog_manager.cc:1348] Generated new cluster ID: 76f258be5d73465183430c82773d9c8c
I20250114 03:02:52.589219 1554 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:02:52.604295 1554 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:02:52.605522 1554 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:02:52.618630 1554 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a5e42d3f5b224492a308f67b4f8006fb: Generated new TSK 0
I20250114 03:02:52.619478 1554 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:02:52.630139 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--builtin_ntp_servers=127.1.109.84:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20250114 03:02:52.737159 1564 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:02:52.737435 1564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:02:52.737478 1564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:02:52.737555 1564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:02:52.740690 1564 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250114 03:02:52.740777 1564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:02:52.740901 1564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:02:52.744714 1564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.1564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:02:52.745877 1564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:02:52.747035 1564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:02:52.755568 1572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:52.756304 1570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:52.756574 1569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:02:52.757508 1564 server_base.cc:1034] running on GCE node
I20250114 03:02:52.757884 1564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:02:52.758440 1564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:02:52.759593 1564 hybrid_clock.cc:648] HybridClock initialized: now 1736823772759575 us; error 29 us; skew 500 ppm
I20250114 03:02:52.761500 1564 webserver.cc:458] Webserver started at http://127.1.109.65:44909/ using document root <none> and password file <none>
I20250114 03:02:52.762069 1564 fs_manager.cc:362] Metadata directory not provided
I20250114 03:02:52.762128 1564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:02:52.762307 1564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:02:52.763971 1564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data/instance:
uuid: "1491b744aa504f8d9f9e2fd0a284d9e1"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.764449 1564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal/instance:
uuid: "1491b744aa504f8d9f9e2fd0a284d9e1"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.767890 1564 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:02:52.770087 1578 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.771044 1564 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:02:52.771150 1564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal
uuid: "1491b744aa504f8d9f9e2fd0a284d9e1"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.771234 1564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:02:52.792168 1564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:02:52.792771 1564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:02:52.792946 1564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:02:52.793475 1564 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:02:52.794381 1564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:02:52.794428 1564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.794485 1564 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:02:52.794520 1564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.802834 1564 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:46245
I20250114 03:02:52.802860 1692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:46245 every 8 connection(s)
I20250114 03:02:52.803751 1564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/data/info.pb
I20250114 03:02:52.805958 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 1564
I20250114 03:02:52.806059 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-0/wal/instance
I20250114 03:02:52.808573 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--builtin_ntp_servers=127.1.109.84:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
I20250114 03:02:52.813580 1693 heartbeater.cc:346] Connected to a master server at 127.1.109.126:37099
I20250114 03:02:52.813814 1693 heartbeater.cc:463] Registering TS with master...
I20250114 03:02:52.814348 1693 heartbeater.cc:510] Master 127.1.109.126:37099 requested a full tablet report, sending...
I20250114 03:02:52.815850 1504 ts_manager.cc:194] Registered new tserver with Master: 1491b744aa504f8d9f9e2fd0a284d9e1 (127.1.109.65:46245)
I20250114 03:02:52.817143 1504 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:41137
W20250114 03:02:52.917776 1696 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:02:52.918062 1696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:02:52.918112 1696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:02:52.918187 1696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:02:52.921291 1696 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250114 03:02:52.921367 1696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:02:52.921469 1696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:02:52.925195 1696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.1696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:02:52.926275 1696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:02:52.927369 1696 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:02:52.935684 1701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:52.936429 1702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:52.936735 1704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:02:52.937589 1696 server_base.cc:1034] running on GCE node
I20250114 03:02:52.937924 1696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:02:52.938412 1696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:02:52.939558 1696 hybrid_clock.cc:648] HybridClock initialized: now 1736823772939534 us; error 34 us; skew 500 ppm
I20250114 03:02:52.941457 1696 webserver.cc:458] Webserver started at http://127.1.109.66:40225/ using document root <none> and password file <none>
I20250114 03:02:52.942075 1696 fs_manager.cc:362] Metadata directory not provided
I20250114 03:02:52.942138 1696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:02:52.942317 1696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:02:52.943933 1696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data/instance:
uuid: "4d0013eaaa1b4a88a710fef184e0e69c"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.944408 1696 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal/instance:
uuid: "4d0013eaaa1b4a88a710fef184e0e69c"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.947695 1696 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:02:52.949731 1710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.950726 1696 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:02:52.950829 1696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal
uuid: "4d0013eaaa1b4a88a710fef184e0e69c"
format_stamp: "Formatted at 2025-01-14 03:02:52 on dist-test-slave-r16g"
I20250114 03:02:52.950915 1696 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:02:52.974601 1696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:02:52.975208 1696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:02:52.975418 1696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:02:52.975931 1696 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:02:52.976771 1696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:02:52.976814 1696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.976894 1696 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:02:52.976917 1696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:52.985530 1696 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:33731
I20250114 03:02:52.985565 1824 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:33731 every 8 connection(s)
I20250114 03:02:52.986399 1696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/data/info.pb
I20250114 03:02:52.994650 1825 heartbeater.cc:346] Connected to a master server at 127.1.109.126:37099
I20250114 03:02:52.994863 1825 heartbeater.cc:463] Registering TS with master...
I20250114 03:02:52.995370 1825 heartbeater.cc:510] Master 127.1.109.126:37099 requested a full tablet report, sending...
I20250114 03:02:52.995548 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 1696
I20250114 03:02:52.995648 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-1/wal/instance
I20250114 03:02:52.996541 1504 ts_manager.cc:194] Registered new tserver with Master: 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731)
I20250114 03:02:52.997172 1504 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:47683
I20250114 03:02:52.998041 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--builtin_ntp_servers=127.1.109.84:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20250114 03:02:53.103338 1828 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:02:53.103592 1828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:02:53.103639 1828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:02:53.103711 1828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:02:53.106782 1828 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250114 03:02:53.106858 1828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:02:53.106946 1828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:02:53.110630 1828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.1828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:02:53.111687 1828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:02:53.112805 1828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:02:53.120291 1833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:53.121007 1834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:53.121788 1836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:02:53.122620 1828 server_base.cc:1034] running on GCE node
I20250114 03:02:53.122983 1828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:02:53.123503 1828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:02:53.124660 1828 hybrid_clock.cc:648] HybridClock initialized: now 1736823773124627 us; error 38 us; skew 500 ppm
I20250114 03:02:53.126610 1828 webserver.cc:458] Webserver started at http://127.1.109.67:41227/ using document root <none> and password file <none>
I20250114 03:02:53.127146 1828 fs_manager.cc:362] Metadata directory not provided
I20250114 03:02:53.127197 1828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:02:53.127341 1828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:02:53.129030 1828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data/instance:
uuid: "1440eae879aa43219b92dd6fc4147e44"
format_stamp: "Formatted at 2025-01-14 03:02:53 on dist-test-slave-r16g"
I20250114 03:02:53.129487 1828 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal/instance:
uuid: "1440eae879aa43219b92dd6fc4147e44"
format_stamp: "Formatted at 2025-01-14 03:02:53 on dist-test-slave-r16g"
I20250114 03:02:53.132755 1828 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250114 03:02:53.134907 1842 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:53.135879 1828 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250114 03:02:53.135983 1828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal
uuid: "1440eae879aa43219b92dd6fc4147e44"
format_stamp: "Formatted at 2025-01-14 03:02:53 on dist-test-slave-r16g"
I20250114 03:02:53.136065 1828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:02:53.149142 1828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:02:53.149765 1828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:02:53.149920 1828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:02:53.150435 1828 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:02:53.151326 1828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:02:53.151376 1828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:53.151430 1828 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:02:53.151468 1828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:53.160025 1828 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:43687
I20250114 03:02:53.160080 1956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:43687 every 8 connection(s)
I20250114 03:02:53.161032 1828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/data/info.pb
I20250114 03:02:53.163491 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 1828
I20250114 03:02:53.163601 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-2/wal/instance
I20250114 03:02:53.169965 1957 heartbeater.cc:346] Connected to a master server at 127.1.109.126:37099
I20250114 03:02:53.170161 1957 heartbeater.cc:463] Registering TS with master...
I20250114 03:02:53.170640 1957 heartbeater.cc:510] Master 127.1.109.126:37099 requested a full tablet report, sending...
I20250114 03:02:53.171533 1504 ts_manager.cc:194] Registered new tserver with Master: 1440eae879aa43219b92dd6fc4147e44 (127.1.109.67:43687)
I20250114 03:02:53.172057 1504 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:59069
I20250114 03:02:53.178006 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:02:53.179625 1461 test_util.cc:274] Using random seed: -871780636
I20250114 03:02:53.193091 1504 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53484:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 03:02:53.194062 1504 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:02:53.208890 1758 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3a5159dbb1424c98fbb271bd682ede (DEFAULT_TABLE table=test-workload [id=94de4aef60ae47da8d2de7a194c031ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:02:53.209158 1627 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3a5159dbb1424c98fbb271bd682ede (DEFAULT_TABLE table=test-workload [id=94de4aef60ae47da8d2de7a194c031ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:02:53.209867 1758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3a5159dbb1424c98fbb271bd682ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:02:53.209908 1627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3a5159dbb1424c98fbb271bd682ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:02:53.209657 1891 tablet_service.cc:1467] Processing CreateTablet for tablet 7b3a5159dbb1424c98fbb271bd682ede (DEFAULT_TABLE table=test-workload [id=94de4aef60ae47da8d2de7a194c031ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:02:53.210410 1891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b3a5159dbb1424c98fbb271bd682ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:02:53.214813 1974 tablet_bootstrap.cc:492] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: Bootstrap starting.
I20250114 03:02:53.216143 1976 tablet_bootstrap.cc:492] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44: Bootstrap starting.
I20250114 03:02:53.216143 1975 tablet_bootstrap.cc:492] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: Bootstrap starting.
I20250114 03:02:53.217360 1974 tablet_bootstrap.cc:654] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: Neither blocks nor log segments found. Creating new log.
I20250114 03:02:53.217998 1974 log.cc:826] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: Log is configured to *not* fsync() on all Append() calls
I20250114 03:02:53.218217 1975 tablet_bootstrap.cc:654] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: Neither blocks nor log segments found. Creating new log.
I20250114 03:02:53.218217 1976 tablet_bootstrap.cc:654] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44: Neither blocks nor log segments found. Creating new log.
I20250114 03:02:53.218973 1976 log.cc:826] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44: Log is configured to *not* fsync() on all Append() calls
I20250114 03:02:53.218973 1975 log.cc:826] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: Log is configured to *not* fsync() on all Append() calls
I20250114 03:02:53.219604 1974 tablet_bootstrap.cc:492] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: No bootstrap required, opened a new log
I20250114 03:02:53.219805 1974 ts_tablet_manager.cc:1397] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:02:53.220517 1976 tablet_bootstrap.cc:492] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44: No bootstrap required, opened a new log
I20250114 03:02:53.220672 1976 ts_tablet_manager.cc:1397] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250114 03:02:53.220956 1975 tablet_bootstrap.cc:492] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: No bootstrap required, opened a new log
I20250114 03:02:53.221122 1975 ts_tablet_manager.cc:1397] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:02:53.223809 1976 raft_consensus.cc:357] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.223840 1974 raft_consensus.cc:357] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.223985 1976 raft_consensus.cc:738] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1440eae879aa43219b92dd6fc4147e44, State: Initialized, Role: FOLLOWER
I20250114 03:02:53.224038 1974 raft_consensus.cc:738] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1491b744aa504f8d9f9e2fd0a284d9e1, State: Initialized, Role: FOLLOWER
I20250114 03:02:53.224273 1976 consensus_queue.cc:260] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.224325 1974 consensus_queue.cc:260] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.224910 1957 heartbeater.cc:502] Master 127.1.109.126:37099 was elected leader, sending a full tablet report...
I20250114 03:02:53.224879 1975 raft_consensus.cc:357] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.225143 1975 raft_consensus.cc:738] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0013eaaa1b4a88a710fef184e0e69c, State: Initialized, Role: FOLLOWER
I20250114 03:02:53.225189 1976 ts_tablet_manager.cc:1428] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44: Time spent starting tablet: real 0.004s user 0.004s sys 0.001s
I20250114 03:02:53.225309 1974 ts_tablet_manager.cc:1428] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250114 03:02:53.225517 1975 consensus_queue.cc:260] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.225662 1693 heartbeater.cc:502] Master 127.1.109.126:37099 was elected leader, sending a full tablet report...
I20250114 03:02:53.226130 1825 heartbeater.cc:502] Master 127.1.109.126:37099 was elected leader, sending a full tablet report...
I20250114 03:02:53.226243 1975 ts_tablet_manager.cc:1428] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20250114 03:02:53.237211 1826 tablet.cc:2367] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:02:53.249189 1819 generic_service.cc:139] {username='slave'} at 127.0.0.1:52446 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20250114 03:02:53.249298 1819 generic_service.cc:175] {username='slave'} at 127.0.0.1:52446 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.1.109.66:33731'
W20250114 03:02:53.250124 1819 generic_service.cc:139] {username='slave'} at 127.0.0.1:52446 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:53.250175 1819 generic_service.cc:175] {username='slave'} at 127.0.0.1:52446 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250114 03:02:53.253753 1951 generic_service.cc:139] {username='slave'} at 127.0.0.1:48510 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20250114 03:02:53.253868 1951 generic_service.cc:175] {username='slave'} at 127.0.0.1:48510 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.1.109.66:33731'
W20250114 03:02:53.254601 1951 generic_service.cc:139] {username='slave'} at 127.0.0.1:48510 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:53.254657 1951 generic_service.cc:175] {username='slave'} at 127.0.0.1:48510 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250114 03:02:53.258213 1687 generic_service.cc:139] {username='slave'} at 127.0.0.1:49998 forcing change of non-runtime-safe flag fail_dns_resolution_hostports
I20250114 03:02:53.258306 1687 generic_service.cc:175] {username='slave'} at 127.0.0.1:49998 changed flags via RPC: fail_dns_resolution_hostports from '' to '127.1.109.66:33731'
W20250114 03:02:53.259052 1687 generic_service.cc:139] {username='slave'} at 127.0.0.1:49998 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:53.259105 1687 generic_service.cc:175] {username='slave'} at 127.0.0.1:49998 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20250114 03:02:53.259835 1911 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b3a5159dbb1424c98fbb271bd682ede"
dest_uuid: "1440eae879aa43219b92dd6fc4147e44"
from {username='slave'} at 127.0.0.1:48498
I20250114 03:02:53.260016 1911 raft_consensus.cc:491] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:02:53.260118 1911 raft_consensus.cc:3054] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:02:53.261090 1911 raft_consensus.cc:513] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.262187 1911 leader_election.cc:290] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [CANDIDATE]: Term 1 election: Requested vote from peers 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731), 1491b744aa504f8d9f9e2fd0a284d9e1 (127.1.109.65:46245)
W20250114 03:02:53.262427 1985 leader_election.cc:336] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731
I20250114 03:02:53.265166 1647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b3a5159dbb1424c98fbb271bd682ede" candidate_uuid: "1440eae879aa43219b92dd6fc4147e44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1"
I20250114 03:02:53.265357 1647 raft_consensus.cc:3054] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:02:53.266271 1647 raft_consensus.cc:2463] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1440eae879aa43219b92dd6fc4147e44 in term 1.
I20250114 03:02:53.266633 1846 leader_election.cc:304] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1440eae879aa43219b92dd6fc4147e44, 1491b744aa504f8d9f9e2fd0a284d9e1; no voters: 4d0013eaaa1b4a88a710fef184e0e69c
I20250114 03:02:53.266878 1980 raft_consensus.cc:2798] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:02:53.267096 1980 raft_consensus.cc:695] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 1 LEADER]: Becoming Leader. State: Replica: 1440eae879aa43219b92dd6fc4147e44, State: Running, Role: LEADER
I20250114 03:02:53.267416 1980 consensus_queue.cc:237] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:53.270974 1502 catalog_manager.cc:5526] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1440eae879aa43219b92dd6fc4147e44 (127.1.109.67). New cstate: current_term: 1 leader_uuid: "1440eae879aa43219b92dd6fc4147e44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } health_report { overall_health: HEALTHY } } }
W20250114 03:02:53.304850 1694 tablet.cc:2367] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:02:53.355046 1985 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Couldn't send request to peer 4d0013eaaa1b4a88a710fef184e0e69c. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731. This is attempt 1: this message will repeat every 5th retry.
I20250114 03:02:53.363282 1980 consensus_queue.cc:1035] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 03:02:53.905311 1985 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Couldn't send request to peer 4d0013eaaa1b4a88a710fef184e0e69c. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731. This is attempt 6: this message will repeat every 5th retry.
W20250114 03:02:54.459311 1985 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Couldn't send request to peer 4d0013eaaa1b4a88a710fef184e0e69c. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731. This is attempt 11: this message will repeat every 5th retry.
W20250114 03:02:54.894562 1985 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Couldn't send request to peer 4d0013eaaa1b4a88a710fef184e0e69c. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731. This is attempt 16: this message will repeat every 5th retry.
W20250114 03:02:55.384845 1985 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Couldn't send request to peer 4d0013eaaa1b4a88a710fef184e0e69c. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731. This is attempt 21: this message will repeat every 5th retry.
W20250114 03:02:55.901175 1985 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 4d0013eaaa1b4a88a710fef184e0e69c (127.1.109.66:33731): Couldn't send request to peer 4d0013eaaa1b4a88a710fef184e0e69c. Status: Network error: failed to refresh physical address: injected DNS resolution failure: 127.1.109.66:33731. This is attempt 26: this message will repeat every 5th retry.
I20250114 03:02:56.295815 1991 consensus_queue.cc:579] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Leader has been unable to successfully communicate with peer 4d0013eaaa1b4a88a710fef184e0e69c for more than 3 seconds (3.028s)
W20250114 03:02:56.298032 1502 catalog_manager.cc:5204] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7b3a5159dbb1424c98fbb271bd682ede with cas_config_opid_index -1: no extra replica candidate found for tablet 7b3a5159dbb1424c98fbb271bd682ede (table test-workload [id=94de4aef60ae47da8d2de7a194c031ca]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (3) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250114 03:02:56.331840 1819 generic_service.cc:139] {username='slave'} at 127.0.0.1:52446 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:56.331943 1819 generic_service.cc:175] {username='slave'} at 127.0.0.1:52446 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20250114 03:02:56.332918 1951 generic_service.cc:139] {username='slave'} at 127.0.0.1:48510 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:56.333003 1951 generic_service.cc:175] {username='slave'} at 127.0.0.1:48510 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
W20250114 03:02:56.333832 1687 generic_service.cc:139] {username='slave'} at 127.0.0.1:49998 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:56.333904 1687 generic_service.cc:175] {username='slave'} at 127.0.0.1:49998 changed flags via RPC: fail_dns_resolution from 'true' to 'false'
I20250114 03:02:56.390877 1778 raft_consensus.cc:3054] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:02:56.487303 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.68:0
--local_ip_for_outbound_sockets=127.1.109.68
--webserver_interface=127.1.109.68
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--builtin_ntp_servers=127.1.109.84:33091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--dns_resolver_cache_capacity_mb=0
--enable_leader_failure_detection=false
--raft_heartbeat_interval_ms=100
--follower_unavailable_considered_failed_sec=3
--leader_failure_max_missed_heartbeat_periods=10
--leader_failure_exp_backoff_max_delta_ms=5000 with env {}
W20250114 03:02:56.593669 1999 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:02:56.593935 1999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:02:56.593981 1999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:02:56.594048 1999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:02:56.597301 1999 flags.cc:425] Enabled experimental flag: --leader_failure_exp_backoff_max_delta_ms=5000
W20250114 03:02:56.597376 1999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:02:56.597467 1999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.68
I20250114 03:02:56.601123 1999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=3
--enable_leader_failure_detection=false
--leader_failure_exp_backoff_max_delta_ms=5000
--leader_failure_max_missed_heartbeat_periods=10
--raft_heartbeat_interval_ms=100
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.1.109.68
--webserver_port=0
--tserver_master_addrs=127.1.109.126:37099
--never_fsync=true
--heap_profile_path=/tmp/kudu.1999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--dns_resolver_cache_capacity_mb=0
--local_ip_for_outbound_sockets=127.1.109.68
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:02:56.602173 1999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:02:56.603267 1999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:02:56.610965 2004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:56.611804 2007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:02:56.612012 2005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:02:56.613004 1999 server_base.cc:1034] running on GCE node
I20250114 03:02:56.613348 1999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:02:56.613893 1999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:02:56.615039 1999 hybrid_clock.cc:648] HybridClock initialized: now 1736823776615028 us; error 23 us; skew 500 ppm
I20250114 03:02:56.622994 1999 webserver.cc:458] Webserver started at http://127.1.109.68:46249/ using document root <none> and password file <none>
I20250114 03:02:56.623566 1999 fs_manager.cc:362] Metadata directory not provided
I20250114 03:02:56.623627 1999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:02:56.623797 1999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:02:56.625442 1999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data/instance:
uuid: "69d19e336a554c66a18d0bde8d0ed2e6"
format_stamp: "Formatted at 2025-01-14 03:02:56 on dist-test-slave-r16g"
I20250114 03:02:56.625910 1999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal/instance:
uuid: "69d19e336a554c66a18d0bde8d0ed2e6"
format_stamp: "Formatted at 2025-01-14 03:02:56 on dist-test-slave-r16g"
I20250114 03:02:56.629323 1999 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250114 03:02:56.631464 2014 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:56.632443 1999 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:02:56.632550 1999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal
uuid: "69d19e336a554c66a18d0bde8d0ed2e6"
format_stamp: "Formatted at 2025-01-14 03:02:56 on dist-test-slave-r16g"
I20250114 03:02:56.632629 1999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:02:56.645886 1999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:02:56.646450 1999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:02:56.646602 1999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:02:56.647197 1999 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:02:56.648110 1999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:02:56.648168 1999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:56.648226 1999 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:02:56.648254 1999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:02:56.656733 1999 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.68:42477
I20250114 03:02:56.656762 2127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.68:42477 every 8 connection(s)
I20250114 03:02:56.657662 1999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/data/info.pb
I20250114 03:02:56.663021 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 1999
I20250114 03:02:56.663137 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0/raft_consensus_election-itest-cluster/ts-3/wal/instance
I20250114 03:02:56.666761 2128 heartbeater.cc:346] Connected to a master server at 127.1.109.126:37099
I20250114 03:02:56.666992 2128 heartbeater.cc:463] Registering TS with master...
I20250114 03:02:56.667515 2128 heartbeater.cc:510] Master 127.1.109.126:37099 requested a full tablet report, sending...
I20250114 03:02:56.668517 1502 ts_manager.cc:194] Registered new tserver with Master: 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68:42477)
I20250114 03:02:56.669155 1502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.68:37825
I20250114 03:02:56.671020 1911 consensus_queue.cc:237] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } peers { permanent_uuid: "69d19e336a554c66a18d0bde8d0ed2e6" member_type: NON_VOTER last_known_addr { host: "127.1.109.68" port: 42477 } attrs { promote: true } }
I20250114 03:02:56.672721 1778 raft_consensus.cc:1270] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 1 FOLLOWER]: Refusing update from remote peer 1440eae879aa43219b92dd6fc4147e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:02:56.672888 1646 raft_consensus.cc:1270] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 1 FOLLOWER]: Refusing update from remote peer 1440eae879aa43219b92dd6fc4147e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:02:56.673290 1993 consensus_queue.cc:1035] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:02:56.673488 1991 consensus_queue.cc:1035] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:02:56.675365 1991 raft_consensus.cc:2949] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } peers { permanent_uuid: "69d19e336a554c66a18d0bde8d0ed2e6" member_type: NON_VOTER last_known_addr { host: "127.1.109.68" port: 42477 } attrs { promote: true } } }
I20250114 03:02:56.675853 1778 raft_consensus.cc:2949] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } peers { permanent_uuid: "69d19e336a554c66a18d0bde8d0ed2e6" member_type: NON_VOTER last_known_addr { host: "127.1.109.68" port: 42477 } attrs { promote: true } } }
I20250114 03:02:56.677311 1491 catalog_manager.cc:5039] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7b3a5159dbb1424c98fbb271bd682ede with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250114 03:02:56.677497 1646 raft_consensus.cc:2949] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } peers { permanent_uuid: "69d19e336a554c66a18d0bde8d0ed2e6" member_type: NON_VOTER last_known_addr { host: "127.1.109.68" port: 42477 } attrs { promote: true } } }
I20250114 03:02:56.678359 1502 catalog_manager.cc:5526] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 reported cstate change: config changed from index -1 to 2, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) added. New cstate: current_term: 1 leader_uuid: "1440eae879aa43219b92dd6fc4147e44" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "69d19e336a554c66a18d0bde8d0ed2e6" member_type: NON_VOTER last_known_addr { host: "127.1.109.68" port: 42477 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250114 03:02:56.682752 1843 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68:42477): Couldn't send request to peer 69d19e336a554c66a18d0bde8d0ed2e6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b3a5159dbb1424c98fbb271bd682ede. This is attempt 1: this message will repeat every 5th retry.
I20250114 03:02:56.682996 1911 consensus_queue.cc:237] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } }
I20250114 03:02:56.684208 1778 raft_consensus.cc:1270] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 1 FOLLOWER]: Refusing update from remote peer 1440eae879aa43219b92dd6fc4147e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 03:02:56.684216 1646 raft_consensus.cc:1270] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 1 FOLLOWER]: Refusing update from remote peer 1440eae879aa43219b92dd6fc4147e44: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 03:02:56.684641 1992 consensus_queue.cc:1035] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 03:02:56.684772 1993 consensus_queue.cc:1035] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 03:02:56.686287 1991 raft_consensus.cc:2949] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } }
I20250114 03:02:56.686784 1646 raft_consensus.cc:2949] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } }
I20250114 03:02:56.686831 1778 raft_consensus.cc:2949] T 7b3a5159dbb1424c98fbb271bd682ede P 4d0013eaaa1b4a88a710fef184e0e69c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } }
I20250114 03:02:56.687741 1491 catalog_manager.cc:5039] ChangeConfig:REMOVE_PEER RPC for tablet 7b3a5159dbb1424c98fbb271bd682ede with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250114 03:02:56.688941 1503 catalog_manager.cc:5526] T 7b3a5159dbb1424c98fbb271bd682ede P 1491b744aa504f8d9f9e2fd0a284d9e1 reported cstate change: config changed from index 2 to 3, NON_VOTER 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68) evicted. New cstate: current_term: 1 leader_uuid: "1440eae879aa43219b92dd6fc4147e44" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4d0013eaaa1b4a88a710fef184e0e69c" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 33731 } } peers { permanent_uuid: "1491b744aa504f8d9f9e2fd0a284d9e1" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46245 } } peers { permanent_uuid: "1440eae879aa43219b92dd6fc4147e44" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 43687 } } }
I20250114 03:02:56.694788 2062 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b3a5159dbb1424c98fbb271bd682ede with delete_type TABLET_DATA_TOMBSTONED (TS 69d19e336a554c66a18d0bde8d0ed2e6 not found in new config with opid_index 3) from {username='slave'} at 127.0.0.1:60044
W20250114 03:02:56.696034 1488 catalog_manager.cc:4839] TS 69d19e336a554c66a18d0bde8d0ed2e6 (127.1.109.68:42477): delete failed for tablet 7b3a5159dbb1424c98fbb271bd682ede because the tablet was not found. No further retry: Not found: Tablet not found: 7b3a5159dbb1424c98fbb271bd682ede
W20250114 03:02:56.716425 1819 generic_service.cc:139] {username='slave'} at 127.0.0.1:52446 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:56.716526 1819 generic_service.cc:175] {username='slave'} at 127.0.0.1:52446 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250114 03:02:56.717383 1951 generic_service.cc:139] {username='slave'} at 127.0.0.1:48510 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:56.717456 1951 generic_service.cc:175] {username='slave'} at 127.0.0.1:48510 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
W20250114 03:02:56.718237 1687 generic_service.cc:139] {username='slave'} at 127.0.0.1:49998 forcing change of non-runtime-safe flag fail_dns_resolution
I20250114 03:02:56.718312 1687 generic_service.cc:175] {username='slave'} at 127.0.0.1:49998 changed flags via RPC: fail_dns_resolution from 'false' to 'true'
I20250114 03:02:56.719044 1911 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b3a5159dbb1424c98fbb271bd682ede"
dest_uuid: "1440eae879aa43219b92dd6fc4147e44"
mode: GRACEFUL
new_leader_uuid: "1440eae879aa43219b92dd6fc4147e44"
from {username='slave'} at 127.0.0.1:48498
I20250114 03:02:56.719218 1911 raft_consensus.cc:604] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 [term 1 LEADER]: Received request to transfer leadership to TS 1440eae879aa43219b92dd6fc4147e44
I20250114 03:02:57.671190 2128 heartbeater.cc:502] Master 127.1.109.126:37099 was elected leader, sending a full tablet report...
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/integration-tests/raft_consensus_election-itest.cc:292: Failure
Value of: tablets.empty()
Actual: true
Expected: false
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/test_util.cc:401: Failure
Failed
Timed out waiting for assertion to pass.
I20250114 03:03:26.796116 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 1564
I20250114 03:03:26.803262 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 1696
W20250114 03:03:26.811051 1846 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.65:46245: connect: Connection refused (error 111)
W20250114 03:03:26.812031 1846 consensus_peers.cc:487] T 7b3a5159dbb1424c98fbb271bd682ede P 1440eae879aa43219b92dd6fc4147e44 -> Peer 1491b744aa504f8d9f9e2fd0a284d9e1 (127.1.109.65:46245): Couldn't send request to peer 1491b744aa504f8d9f9e2fd0a284d9e1. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.65:46245: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:03:26.812425 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 1828
I20250114 03:03:26.819497 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 1999
I20250114 03:03:26.825992 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 1473
2025-01-14T03:03:26Z chronyd exiting
I20250114 03:03:26.844421 1461 test_util.cc:181] -----------------------------------------------
I20250114 03:03:26.844482 1461 test_util.cc:182] Had failures, leaving test files at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TestNewLeaderCantResolvePeers.1736823772370706-1461-0
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers (34464 ms)
[ RUN ] RaftConsensusElectionITest.RunLeaderElection
I20250114 03:03:26.845227 1461 test_util.cc:274] Using random seed: -838115033
I20250114 03:03:26.845965 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:03:26.846004 1461 ts_itest-base.cc:113] --------------
I20250114 03:03:26.846022 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:03:26.846038 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:03:26.846065 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:03:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:03:26Z Disabled control of system clock
I20250114 03:03:26.854132 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34373
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:34373 with env {}
W20250114 03:03:26.967051 2160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:26.967294 2160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:26.967336 2160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:26.970670 2160 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:03:26.970758 2160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:26.970789 2160 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:03:26.970849 2160 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:03:26.974823 2160 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:34373
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:26.976023 2160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:26.977268 2160 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:26.985280 2168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:26.985996 2166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:26.986680 2165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:26.987861 2160 server_base.cc:1034] running on GCE node
I20250114 03:03:26.988380 2160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:26.989413 2160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:26.990566 2160 hybrid_clock.cc:648] HybridClock initialized: now 1736823806990549 us; error 31 us; skew 500 ppm
I20250114 03:03:26.996330 2160 webserver.cc:458] Webserver started at http://127.1.109.126:38047/ using document root <none> and password file <none>
I20250114 03:03:26.997010 2160 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:26.997078 2160 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:26.997275 2160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:26.999055 2160 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "345ad9d6dabd4f739067c1fa67a6bdef"
format_stamp: "Formatted at 2025-01-14 03:03:26 on dist-test-slave-r16g"
I20250114 03:03:26.999524 2160 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "345ad9d6dabd4f739067c1fa67a6bdef"
format_stamp: "Formatted at 2025-01-14 03:03:26 on dist-test-slave-r16g"
I20250114 03:03:27.003070 2160 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:03:27.005288 2174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.006350 2160 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:03:27.006474 2160 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "345ad9d6dabd4f739067c1fa67a6bdef"
format_stamp: "Formatted at 2025-01-14 03:03:26 on dist-test-slave-r16g"
I20250114 03:03:27.006556 2160 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:27.018285 2160 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:27.018993 2160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:27.019155 2160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:27.026176 2160 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:34373
I20250114 03:03:27.026219 2226 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:34373 every 8 connection(s)
I20250114 03:03:27.027176 2160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:03:27.029368 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 2160
I20250114 03:03:27.029476 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:03:27.030130 2227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:27.037240 2227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef: Bootstrap starting.
I20250114 03:03:27.039642 2227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:27.040365 2227 log.cc:826] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:27.042111 2227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef: No bootstrap required, opened a new log
I20250114 03:03:27.045220 2227 raft_consensus.cc:357] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34373 } }
I20250114 03:03:27.045428 2227 raft_consensus.cc:383] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:27.045477 2227 raft_consensus.cc:738] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 345ad9d6dabd4f739067c1fa67a6bdef, State: Initialized, Role: FOLLOWER
I20250114 03:03:27.045909 2227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34373 } }
I20250114 03:03:27.046025 2227 raft_consensus.cc:397] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:03:27.046088 2227 raft_consensus.cc:491] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:03:27.046216 2227 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:27.047108 2227 raft_consensus.cc:513] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34373 } }
I20250114 03:03:27.047462 2227 leader_election.cc:304] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 345ad9d6dabd4f739067c1fa67a6bdef; no voters:
I20250114 03:03:27.047762 2227 leader_election.cc:290] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:03:27.047868 2232 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:03:27.048106 2232 raft_consensus.cc:695] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [term 1 LEADER]: Becoming Leader. State: Replica: 345ad9d6dabd4f739067c1fa67a6bdef, State: Running, Role: LEADER
I20250114 03:03:27.048417 2232 consensus_queue.cc:237] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34373 } }
I20250114 03:03:27.049156 2227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:03:27.050550 2233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34373 } } }
I20250114 03:03:27.050562 2234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 345ad9d6dabd4f739067c1fa67a6bdef. Latest consensus state: current_term: 1 leader_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "345ad9d6dabd4f739067c1fa67a6bdef" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34373 } } }
I20250114 03:03:27.050756 2234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [sys.catalog]: This master's current role is: LEADER
I20250114 03:03:27.051035 2233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef [sys.catalog]: This master's current role is: LEADER
W20250114 03:03:27.053882 2247 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 03:03:27.053972 2247 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 03:03:27.054232 2243 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:03:27.055260 2243 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:03:27.060699 2243 catalog_manager.cc:1348] Generated new cluster ID: 5b51d0907b3e4bd78759568c671fa56b
I20250114 03:03:27.060776 2243 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:03:27.077715 2243 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:03:27.078697 2243 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:03:27.093506 2243 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 345ad9d6dabd4f739067c1fa67a6bdef: Generated new TSK 0
I20250114 03:03:27.094329 2243 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:03:27.106057 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34373
--builtin_ntp_servers=127.1.109.84:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20250114 03:03:27.214672 2251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:27.214900 2251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:27.214979 2251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:27.218115 2251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:27.218250 2251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:03:27.222057 2251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34373
--never_fsync=true
--heap_profile_path=/tmp/kudu.2251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:27.223166 2251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:27.224298 2251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:27.232647 2257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:27.233567 2256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:27.234251 2259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:27.235057 2251 server_base.cc:1034] running on GCE node
I20250114 03:03:27.235427 2251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:27.236019 2251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:27.237165 2251 hybrid_clock.cc:648] HybridClock initialized: now 1736823807237153 us; error 28 us; skew 500 ppm
I20250114 03:03:27.239073 2251 webserver.cc:458] Webserver started at http://127.1.109.65:41721/ using document root <none> and password file <none>
I20250114 03:03:27.239661 2251 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:27.239722 2251 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:27.239903 2251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:27.241632 2251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0b918a3d1f5a4d03859ea5200d2f8c12"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.242125 2251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0b918a3d1f5a4d03859ea5200d2f8c12"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.245575 2251 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:03:27.247841 2265 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.248924 2251 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:03:27.249034 2251 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0b918a3d1f5a4d03859ea5200d2f8c12"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.249126 2251 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:27.268131 2251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:27.268816 2251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:27.269018 2251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:27.269582 2251 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:27.270576 2251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:03:27.270637 2251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.270689 2251 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:03:27.270718 2251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.279793 2251 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:33989
I20250114 03:03:27.279824 2379 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:33989 every 8 connection(s)
I20250114 03:03:27.280774 2251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:03:27.282275 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 2251
I20250114 03:03:27.282392 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:03:27.284795 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34373
--builtin_ntp_servers=127.1.109.84:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20250114 03:03:27.291216 2380 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34373
I20250114 03:03:27.291509 2380 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:27.292269 2380 heartbeater.cc:510] Master 127.1.109.126:34373 requested a full tablet report, sending...
I20250114 03:03:27.293903 2191 ts_manager.cc:194] Registered new tserver with Master: 0b918a3d1f5a4d03859ea5200d2f8c12 (127.1.109.65:33989)
I20250114 03:03:27.295114 2191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:54747
W20250114 03:03:27.398053 2383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:27.398303 2383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:27.398389 2383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:27.401459 2383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:27.401588 2383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:03:27.405402 2383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34373
--never_fsync=true
--heap_profile_path=/tmp/kudu.2383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:27.406541 2383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:27.407692 2383 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:27.416646 2388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:27.417237 2391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:27.417881 2389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:27.419045 2383 server_base.cc:1034] running on GCE node
I20250114 03:03:27.419438 2383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:27.420038 2383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:27.421185 2383 hybrid_clock.cc:648] HybridClock initialized: now 1736823807421149 us; error 52 us; skew 500 ppm
I20250114 03:03:27.423126 2383 webserver.cc:458] Webserver started at http://127.1.109.66:37357/ using document root <none> and password file <none>
I20250114 03:03:27.423707 2383 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:27.423766 2383 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:27.423947 2383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:27.425730 2383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "182afcb279e24f81bfc26919a0739481"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.426215 2383 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "182afcb279e24f81bfc26919a0739481"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.429700 2383 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:03:27.431951 2397 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.433020 2383 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:03:27.433127 2383 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "182afcb279e24f81bfc26919a0739481"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.433214 2383 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:27.448053 2383 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:27.448732 2383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:27.448935 2383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:27.449609 2383 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:27.450608 2383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:03:27.450657 2383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.450719 2383 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:03:27.450767 2383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.459769 2383 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:41503
I20250114 03:03:27.459836 2511 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:41503 every 8 connection(s)
I20250114 03:03:27.460891 2383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:03:27.461089 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 2383
I20250114 03:03:27.461194 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:03:27.463732 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34373
--builtin_ntp_servers=127.1.109.84:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
I20250114 03:03:27.470813 2512 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34373
I20250114 03:03:27.471060 2512 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:27.471634 2512 heartbeater.cc:510] Master 127.1.109.126:34373 requested a full tablet report, sending...
I20250114 03:03:27.472719 2191 ts_manager.cc:194] Registered new tserver with Master: 182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503)
I20250114 03:03:27.473317 2191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:43655
W20250114 03:03:27.572661 2515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:27.572937 2515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:27.572990 2515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:27.576057 2515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:27.576162 2515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:03:27.579950 2515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34373
--never_fsync=true
--heap_profile_path=/tmp/kudu.2515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:27.581115 2515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:27.582301 2515 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:27.591352 2520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:27.592116 2523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:27.592952 2521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:27.593747 2515 server_base.cc:1034] running on GCE node
I20250114 03:03:27.594141 2515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:27.594731 2515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:27.595889 2515 hybrid_clock.cc:648] HybridClock initialized: now 1736823807595855 us; error 46 us; skew 500 ppm
I20250114 03:03:27.597820 2515 webserver.cc:458] Webserver started at http://127.1.109.67:39033/ using document root <none> and password file <none>
I20250114 03:03:27.598392 2515 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:27.598448 2515 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:27.598628 2515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:27.600302 2515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "4be6a2a583bf404c807ec84a64d39555"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.600733 2515 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "4be6a2a583bf404c807ec84a64d39555"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.604130 2515 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250114 03:03:27.606379 2529 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.607395 2515 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:03:27.607491 2515 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "4be6a2a583bf404c807ec84a64d39555"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:27.607595 2515 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:27.630304 2515 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:27.631031 2515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:27.631194 2515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:27.631783 2515 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:27.632719 2515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:03:27.632767 2515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.632815 2515 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:03:27.632867 2515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:27.641608 2515 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:45871
I20250114 03:03:27.641621 2643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:45871 every 8 connection(s)
I20250114 03:03:27.642580 2515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:03:27.649681 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 2515
I20250114 03:03:27.649784 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:03:27.651859 2644 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34373
I20250114 03:03:27.652096 2644 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:27.652678 2644 heartbeater.cc:510] Master 127.1.109.126:34373 requested a full tablet report, sending...
I20250114 03:03:27.653699 2191 ts_manager.cc:194] Registered new tserver with Master: 4be6a2a583bf404c807ec84a64d39555 (127.1.109.67:45871)
I20250114 03:03:27.654300 2191 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:35065
I20250114 03:03:27.664459 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:03:27.673452 2191 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37024:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250114 03:03:27.676461 2191 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:03:27.693271 2313 tablet_service.cc:1467] Processing CreateTablet for tablet 82fd6ee9a55342ef9a8c38d3a41cd70e (DEFAULT_TABLE table=TestTable [id=08d5632dffee4ba6b185269a3086b3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:03:27.693276 2578 tablet_service.cc:1467] Processing CreateTablet for tablet 82fd6ee9a55342ef9a8c38d3a41cd70e (DEFAULT_TABLE table=TestTable [id=08d5632dffee4ba6b185269a3086b3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:03:27.693276 2446 tablet_service.cc:1467] Processing CreateTablet for tablet 82fd6ee9a55342ef9a8c38d3a41cd70e (DEFAULT_TABLE table=TestTable [id=08d5632dffee4ba6b185269a3086b3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:03:27.694211 2313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82fd6ee9a55342ef9a8c38d3a41cd70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:27.694283 2578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82fd6ee9a55342ef9a8c38d3a41cd70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:27.694283 2446 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82fd6ee9a55342ef9a8c38d3a41cd70e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:27.699721 2661 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: Bootstrap starting.
I20250114 03:03:27.701181 2663 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Bootstrap starting.
I20250114 03:03:27.701201 2662 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: Bootstrap starting.
I20250114 03:03:27.702701 2661 tablet_bootstrap.cc:654] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:27.703051 2663 tablet_bootstrap.cc:654] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:27.703250 2662 tablet_bootstrap.cc:654] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:27.703625 2661 log.cc:826] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:27.703742 2663 log.cc:826] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:27.703995 2662 log.cc:826] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:27.705740 2663 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: No bootstrap required, opened a new log
I20250114 03:03:27.705948 2663 ts_tablet_manager.cc:1397] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:03:27.706061 2661 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: No bootstrap required, opened a new log
I20250114 03:03:27.706233 2661 ts_tablet_manager.cc:1397] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20250114 03:03:27.706269 2662 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: No bootstrap required, opened a new log
I20250114 03:03:27.708191 2662 ts_tablet_manager.cc:1397] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20250114 03:03:27.710151 2663 raft_consensus.cc:357] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.710381 2663 raft_consensus.cc:383] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:27.710438 2663 raft_consensus.cc:738] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182afcb279e24f81bfc26919a0739481, State: Initialized, Role: FOLLOWER
I20250114 03:03:27.710459 2661 raft_consensus.cc:357] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.710666 2661 raft_consensus.cc:383] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:27.710728 2661 raft_consensus.cc:738] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b918a3d1f5a4d03859ea5200d2f8c12, State: Initialized, Role: FOLLOWER
I20250114 03:03:27.710924 2663 consensus_queue.cc:260] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.711325 2661 consensus_queue.cc:260] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.712057 2663 ts_tablet_manager.cc:1428] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20250114 03:03:27.712478 2512 heartbeater.cc:502] Master 127.1.109.126:34373 was elected leader, sending a full tablet report...
I20250114 03:03:27.713109 2661 ts_tablet_manager.cc:1428] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 03:03:27.713400 2380 heartbeater.cc:502] Master 127.1.109.126:34373 was elected leader, sending a full tablet report...
I20250114 03:03:27.713861 2662 raft_consensus.cc:357] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.714166 2662 raft_consensus.cc:383] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:27.714270 2662 raft_consensus.cc:738] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4be6a2a583bf404c807ec84a64d39555, State: Initialized, Role: FOLLOWER
I20250114 03:03:27.715070 2662 consensus_queue.cc:260] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.715828 2644 heartbeater.cc:502] Master 127.1.109.126:34373 was elected leader, sending a full tablet report...
I20250114 03:03:27.715997 2662 ts_tablet_manager.cc:1428] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20250114 03:03:27.781955 2381 tablet.cc:2367] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:03:27.893541 2645 tablet.cc:2367] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:03:27.962172 2513 tablet.cc:2367] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:03:27.983776 2667 raft_consensus.cc:491] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:03:27.983994 2667 raft_consensus.cc:513] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.985209 2667 leader_election.cc:290] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b918a3d1f5a4d03859ea5200d2f8c12 (127.1.109.65:33989), 4be6a2a583bf404c807ec84a64d39555 (127.1.109.67:45871)
I20250114 03:03:27.989159 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fd6ee9a55342ef9a8c38d3a41cd70e" candidate_uuid: "182afcb279e24f81bfc26919a0739481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be6a2a583bf404c807ec84a64d39555" is_pre_election: true
I20250114 03:03:27.989230 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fd6ee9a55342ef9a8c38d3a41cd70e" candidate_uuid: "182afcb279e24f81bfc26919a0739481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" is_pre_election: true
I20250114 03:03:27.989440 2598 raft_consensus.cc:2463] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182afcb279e24f81bfc26919a0739481 in term 0.
I20250114 03:03:27.989478 2333 raft_consensus.cc:2463] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 182afcb279e24f81bfc26919a0739481 in term 0.
I20250114 03:03:27.989878 2400 leader_election.cc:304] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b918a3d1f5a4d03859ea5200d2f8c12, 182afcb279e24f81bfc26919a0739481; no voters:
I20250114 03:03:27.990134 2667 raft_consensus.cc:2798] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:03:27.990207 2667 raft_consensus.cc:491] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:03:27.990247 2667 raft_consensus.cc:3054] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:27.991209 2667 raft_consensus.cc:513] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.991601 2667 leader_election.cc:290] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [CANDIDATE]: Term 1 election: Requested vote from peers 0b918a3d1f5a4d03859ea5200d2f8c12 (127.1.109.65:33989), 4be6a2a583bf404c807ec84a64d39555 (127.1.109.67:45871)
I20250114 03:03:27.991938 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fd6ee9a55342ef9a8c38d3a41cd70e" candidate_uuid: "182afcb279e24f81bfc26919a0739481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12"
I20250114 03:03:27.991952 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fd6ee9a55342ef9a8c38d3a41cd70e" candidate_uuid: "182afcb279e24f81bfc26919a0739481" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4be6a2a583bf404c807ec84a64d39555"
I20250114 03:03:27.992045 2333 raft_consensus.cc:3054] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:27.992091 2598 raft_consensus.cc:3054] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:27.992990 2333 raft_consensus.cc:2463] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182afcb279e24f81bfc26919a0739481 in term 1.
I20250114 03:03:27.993041 2598 raft_consensus.cc:2463] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 182afcb279e24f81bfc26919a0739481 in term 1.
I20250114 03:03:27.993273 2400 leader_election.cc:304] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b918a3d1f5a4d03859ea5200d2f8c12, 182afcb279e24f81bfc26919a0739481; no voters:
I20250114 03:03:27.993445 2667 raft_consensus.cc:2798] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:03:27.993666 2667 raft_consensus.cc:695] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 1 LEADER]: Becoming Leader. State: Replica: 182afcb279e24f81bfc26919a0739481, State: Running, Role: LEADER
I20250114 03:03:27.993968 2667 consensus_queue.cc:237] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:27.997013 2190 catalog_manager.cc:5526] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 reported cstate change: term changed from 0 to 1, leader changed from <none> to 182afcb279e24f81bfc26919a0739481 (127.1.109.66). New cstate: current_term: 1 leader_uuid: "182afcb279e24f81bfc26919a0739481" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } health_report { overall_health: HEALTHY } } }
I20250114 03:03:28.052811 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:03:28.054831 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0b918a3d1f5a4d03859ea5200d2f8c12 to finish bootstrapping
I20250114 03:03:28.060882 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 182afcb279e24f81bfc26919a0739481 to finish bootstrapping
I20250114 03:03:28.065737 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4be6a2a583bf404c807ec84a64d39555 to finish bootstrapping
I20250114 03:03:28.090711 2598 raft_consensus.cc:1270] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 1 FOLLOWER]: Refusing update from remote peer 182afcb279e24f81bfc26919a0739481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:03:28.090711 2333 raft_consensus.cc:1270] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 1 FOLLOWER]: Refusing update from remote peer 182afcb279e24f81bfc26919a0739481: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:03:28.091352 2672 consensus_queue.cc:1035] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:03:28.091524 2667 consensus_queue.cc:1035] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:03:28.101361 2677 mvcc.cc:204] Tried to move back new op lower bound from 7114030317932728320 to 7114030317545762816. Current Snapshot: MvccSnapshot[applied={T|T < 7114030317932728320}]
I20250114 03:03:28.196373 2313 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:03:28.196767 2446 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:03:28.197554 2578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
345ad9d6dabd4f739067c1fa67a6bdef | 127.1.109.126:34373 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:39395 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0b918a3d1f5a4d03859ea5200d2f8c12 | 127.1.109.65:33989 | HEALTHY | <none> | 0 | 0
182afcb279e24f81bfc26919a0739481 | 127.1.109.66:41503 | HEALTHY | <none> | 1 | 0
4be6a2a583bf404c807ec84a64d39555 | 127.1.109.67:45871 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:33989
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:41503
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:45871
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:33989
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:41503
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:45871
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:39395 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250114 03:03:28.223800 1461 log_verifier.cc:126] Checking tablet 82fd6ee9a55342ef9a8c38d3a41cd70e
I20250114 03:03:28.251382 1461 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 82fd6ee9a55342ef9a8c38d3a41cd70e
I20250114 03:03:28.287045 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 2383
I20250114 03:03:28.296386 2333 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82fd6ee9a55342ef9a8c38d3a41cd70e"
dest_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12"
from {username='slave'} at 127.0.0.1:34026
I20250114 03:03:28.296667 2333 raft_consensus.cc:491] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:03:28.296748 2333 raft_consensus.cc:3054] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:03:28.297652 2333 raft_consensus.cc:513] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:28.298677 2333 leader_election.cc:290] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [CANDIDATE]: Term 2 election: Requested vote from peers 4be6a2a583bf404c807ec84a64d39555 (127.1.109.67:45871), 182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503)
W20250114 03:03:28.299808 2266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.66:41503: connect: Connection refused (error 111)
W20250114 03:03:28.300889 2266 leader_election.cc:336] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503): Network error: Client connection negotiation failed: client connection to 127.1.109.66:41503: connect: Connection refused (error 111)
I20250114 03:03:28.303009 2598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82fd6ee9a55342ef9a8c38d3a41cd70e" candidate_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: true dest_uuid: "4be6a2a583bf404c807ec84a64d39555"
I20250114 03:03:28.303153 2598 raft_consensus.cc:3054] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:03:28.304023 2598 raft_consensus.cc:2463] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b918a3d1f5a4d03859ea5200d2f8c12 in term 2.
I20250114 03:03:28.304383 2269 leader_election.cc:304] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b918a3d1f5a4d03859ea5200d2f8c12, 4be6a2a583bf404c807ec84a64d39555; no voters: 182afcb279e24f81bfc26919a0739481
I20250114 03:03:28.304711 2668 raft_consensus.cc:2798] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:03:28.304970 2668 raft_consensus.cc:695] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [term 2 LEADER]: Becoming Leader. State: Replica: 0b918a3d1f5a4d03859ea5200d2f8c12, State: Running, Role: LEADER
W20250114 03:03:28.305241 2647 meta_cache.cc:302] tablet 82fd6ee9a55342ef9a8c38d3a41cd70e: replica 182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503) has failed: Network error: Client connection negotiation failed: client connection to 127.1.109.66:41503: connect: Connection refused (error 111)
I20250114 03:03:28.305282 2668 consensus_queue.cc:237] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:28.307472 2189 catalog_manager.cc:5526] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 reported cstate change: term changed from 1 to 2, leader changed from 182afcb279e24f81bfc26919a0739481 (127.1.109.66) to 0b918a3d1f5a4d03859ea5200d2f8c12 (127.1.109.65). New cstate: current_term: 2 leader_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } health_report { overall_health: UNKNOWN } } }
I20250114 03:03:28.315198 2598 raft_consensus.cc:1270] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 4be6a2a583bf404c807ec84a64d39555 [term 2 FOLLOWER]: Refusing update from remote peer 0b918a3d1f5a4d03859ea5200d2f8c12: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250114 03:03:28.315708 2719 consensus_queue.cc:1035] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250114 03:03:28.315901 2266 consensus_peers.cc:487] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 0b918a3d1f5a4d03859ea5200d2f8c12 -> Peer 182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503): Couldn't send request to peer 182afcb279e24f81bfc26919a0739481. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:41503: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:03:28.351941 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:41503
--local_ip_for_outbound_sockets=127.1.109.66
--tserver_master_addrs=127.1.109.126:34373
--webserver_port=37357
--webserver_interface=127.1.109.66
--builtin_ntp_servers=127.1.109.84:39395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=1000 with env {}
W20250114 03:03:28.458493 2721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:28.458726 2721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:28.458772 2721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:28.461889 2721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:28.461998 2721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:03:28.465698 2721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:39395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=1000
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:41503
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=37357
--tserver_master_addrs=127.1.109.126:34373
--never_fsync=true
--heap_profile_path=/tmp/kudu.2721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:28.466794 2721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:28.467942 2721 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:28.476374 2726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:28.477149 2727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:28.477955 2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:28.478793 2721 server_base.cc:1034] running on GCE node
I20250114 03:03:28.479175 2721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:28.479774 2721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:28.480875 2721 hybrid_clock.cc:648] HybridClock initialized: now 1736823808480854 us; error 35 us; skew 500 ppm
I20250114 03:03:28.482781 2721 webserver.cc:458] Webserver started at http://127.1.109.66:37357/ using document root <none> and password file <none>
I20250114 03:03:28.483422 2721 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:28.483494 2721 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:28.486945 2721 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:03:28.489065 2735 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:28.490141 2721 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:03:28.490244 2721 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "182afcb279e24f81bfc26919a0739481"
format_stamp: "Formatted at 2025-01-14 03:03:27 on dist-test-slave-r16g"
I20250114 03:03:28.490675 2721 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:28.513248 2721 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:28.513947 2721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:28.514120 2721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:28.514704 2721 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:28.515828 2742 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 03:03:28.518959 2721 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:03:28.519012 2721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20250114 03:03:28.519068 2721 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 03:03:28.520957 2721 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:03:28.521000 2721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20250114 03:03:28.521289 2742 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Bootstrap starting.
I20250114 03:03:28.530539 2721 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:41503
I20250114 03:03:28.530611 2850 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:41503 every 8 connection(s)
I20250114 03:03:28.531558 2721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:03:28.533958 2742 log.cc:826] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:28.538057 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 2721
I20250114 03:03:28.540336 2851 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34373
I20250114 03:03:28.540560 2851 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:28.541224 2851 heartbeater.cc:510] Master 127.1.109.126:34373 requested a full tablet report, sending...
I20250114 03:03:28.543488 2189 ts_manager.cc:194] Re-registered known tserver with Master: 182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503)
I20250114 03:03:28.544461 2189 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:34821
I20250114 03:03:28.554020 2742 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=500 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:03:28.554423 2742 tablet_bootstrap.cc:492] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Bootstrap complete.
I20250114 03:03:28.555452 2742 ts_tablet_manager.cc:1397] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Time spent bootstrapping tablet: real 0.034s user 0.027s sys 0.004s
I20250114 03:03:28.557495 2742 raft_consensus.cc:357] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:28.557855 2742 raft_consensus.cc:738] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 182afcb279e24f81bfc26919a0739481, State: Initialized, Role: FOLLOWER
I20250114 03:03:28.558320 2742 consensus_queue.cc:260] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b918a3d1f5a4d03859ea5200d2f8c12" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 33989 } } peers { permanent_uuid: "4be6a2a583bf404c807ec84a64d39555" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 45871 } } peers { permanent_uuid: "182afcb279e24f81bfc26919a0739481" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 41503 } }
I20250114 03:03:28.559031 2851 heartbeater.cc:502] Master 127.1.109.126:34373 was elected leader, sending a full tablet report...
I20250114 03:03:28.559212 2742 ts_tablet_manager.cc:1428] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481: Time spent starting tablet: real 0.004s user 0.006s sys 0.000s
I20250114 03:03:28.587319 2578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:03:28.587404 2313 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:03:28.591150 2784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
345ad9d6dabd4f739067c1fa67a6bdef | 127.1.109.126:34373 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:39395 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0b918a3d1f5a4d03859ea5200d2f8c12 | 127.1.109.65:33989 | HEALTHY | <none> | 1 | 0
182afcb279e24f81bfc26919a0739481 | 127.1.109.66:41503 | HEALTHY | <none> | 0 | 0
4be6a2a583bf404c807ec84a64d39555 | 127.1.109.67:45871 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:33989
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:41503
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:45871
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:33989
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:41503
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:45871
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:39395 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 82fd6ee9a55342ef9a8c38d3a41cd70e of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
0b918a3d1f5a4d03859ea5200d2f8c12 (127.1.109.65:33989): RUNNING [LEADER]
4be6a2a583bf404c807ec84a64d39555 (127.1.109.67:45871): RUNNING
182afcb279e24f81bfc26919a0739481 (127.1.109.66:41503): RUNNING
All reported replicas are:
A = 0b918a3d1f5a4d03859ea5200d2f8c12
B = 4be6a2a583bf404c807ec84a64d39555
C = 182afcb279e24f81bfc26919a0739481
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C | | | Yes
A | A* B C | 2 | -1 | Yes
B | A* B C | 2 | -1 | Yes
C | A B C | 1 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250114 03:03:28.613986 1461 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250114 03:03:28.720960 2804 raft_consensus.cc:3054] T 82fd6ee9a55342ef9a8c38d3a41cd70e P 182afcb279e24f81bfc26919a0739481 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:03:28.730381 2880 mvcc.cc:204] Tried to move back new op lower bound from 7114030318991400960 to 7114030318820921344. Current Snapshot: MvccSnapshot[applied={T|T < 7114030318926426112}]
I20250114 03:03:28.812019 2313 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:03:28.812537 2578 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:03:28.813300 2784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
345ad9d6dabd4f739067c1fa67a6bdef | 127.1.109.126:34373 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:39395 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0b918a3d1f5a4d03859ea5200d2f8c12 | 127.1.109.65:33989 | HEALTHY | <none> | 1 | 0
182afcb279e24f81bfc26919a0739481 | 127.1.109.66:41503 | HEALTHY | <none> | 0 | 0
4be6a2a583bf404c807ec84a64d39555 | 127.1.109.67:45871 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:33989
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:41503
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:45871
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:33989
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:41503
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.RunLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:45871
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:39395 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250114 03:03:28.839304 1461 log_verifier.cc:126] Checking tablet 82fd6ee9a55342ef9a8c38d3a41cd70e
I20250114 03:03:28.861955 1461 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 82fd6ee9a55342ef9a8c38d3a41cd70e
I20250114 03:03:29.263216 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 2251
I20250114 03:03:29.270159 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 2721
I20250114 03:03:29.276203 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 2515
I20250114 03:03:29.282063 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 2160
2025-01-14T03:03:29Z chronyd exiting
[ OK ] RaftConsensusElectionITest.RunLeaderElection (2461 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElections
I20250114 03:03:29.307160 1461 test_util.cc:274] Using random seed: -835653103
I20250114 03:03:29.307965 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:03:29.308009 1461 ts_itest-base.cc:113] --------------
I20250114 03:03:29.308029 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:03:29.308046 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:03:29.308073 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:03:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:03:29Z Disabled control of system clock
I20250114 03:03:29.316341 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:41883
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:41883 with env {}
W20250114 03:03:29.424464 2923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:29.424743 2923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:29.424856 2923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:29.428031 2923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:03:29.428102 2923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:29.428135 2923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:03:29.428166 2923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:03:29.432062 2923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:41883
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:41883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.2923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:29.433308 2923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:29.434471 2923 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:29.442466 2929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:29.443145 2928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:29.444070 2931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:29.444984 2923 server_base.cc:1034] running on GCE node
I20250114 03:03:29.445569 2923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:29.446544 2923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:29.447726 2923 hybrid_clock.cc:648] HybridClock initialized: now 1736823809447703 us; error 41 us; skew 500 ppm
I20250114 03:03:29.449760 2923 webserver.cc:458] Webserver started at http://127.1.109.126:42003/ using document root <none> and password file <none>
I20250114 03:03:29.450341 2923 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:29.450402 2923 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:29.450588 2923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:29.452365 2923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0ed628867e0c43889eff4cd01c13adec"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.452967 2923 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0ed628867e0c43889eff4cd01c13adec"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.456344 2923 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250114 03:03:29.458621 2937 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.459791 2923 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250114 03:03:29.459914 2923 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0ed628867e0c43889eff4cd01c13adec"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.460007 2923 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:29.476976 2923 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:29.477700 2923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:29.477885 2923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:29.484865 2923 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:41883
I20250114 03:03:29.484930 2989 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:41883 every 8 connection(s)
I20250114 03:03:29.485919 2923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:03:29.488559 2990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:29.491756 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 2923
I20250114 03:03:29.491881 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:03:29.494203 2990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec: Bootstrap starting.
I20250114 03:03:29.496346 2990 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:29.497082 2990 log.cc:826] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:29.498965 2990 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec: No bootstrap required, opened a new log
I20250114 03:03:29.502030 2990 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed628867e0c43889eff4cd01c13adec" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 41883 } }
I20250114 03:03:29.502216 2990 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:29.502250 2990 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ed628867e0c43889eff4cd01c13adec, State: Initialized, Role: FOLLOWER
I20250114 03:03:29.502599 2990 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed628867e0c43889eff4cd01c13adec" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 41883 } }
I20250114 03:03:29.502701 2990 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:03:29.502743 2990 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:03:29.502821 2990 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:29.503530 2990 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed628867e0c43889eff4cd01c13adec" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 41883 } }
I20250114 03:03:29.503821 2990 leader_election.cc:304] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ed628867e0c43889eff4cd01c13adec; no voters:
I20250114 03:03:29.504102 2990 leader_election.cc:290] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:03:29.504195 2995 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:03:29.504418 2995 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [term 1 LEADER]: Becoming Leader. State: Replica: 0ed628867e0c43889eff4cd01c13adec, State: Running, Role: LEADER
I20250114 03:03:29.504757 2995 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed628867e0c43889eff4cd01c13adec" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 41883 } }
I20250114 03:03:29.505322 2990 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:03:29.506546 2997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ed628867e0c43889eff4cd01c13adec. Latest consensus state: current_term: 1 leader_uuid: "0ed628867e0c43889eff4cd01c13adec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed628867e0c43889eff4cd01c13adec" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 41883 } } }
I20250114 03:03:29.506685 2997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [sys.catalog]: This master's current role is: LEADER
I20250114 03:03:29.506572 2996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ed628867e0c43889eff4cd01c13adec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ed628867e0c43889eff4cd01c13adec" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 41883 } } }
I20250114 03:03:29.507001 2996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec [sys.catalog]: This master's current role is: LEADER
I20250114 03:03:29.507032 3004 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:03:29.510239 3004 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:03:29.515534 3004 catalog_manager.cc:1348] Generated new cluster ID: fca2a7a05613476298f1d5b5d665c6e8
I20250114 03:03:29.515609 3004 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:03:29.535799 3004 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:03:29.537169 3004 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:03:29.556131 3004 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0ed628867e0c43889eff4cd01c13adec: Generated new TSK 0
I20250114 03:03:29.557176 3004 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:03:29.571153 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:41883
--builtin_ntp_servers=127.1.109.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20250114 03:03:29.679212 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:29.679427 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:29.679481 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:29.682709 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:29.682829 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:03:29.686584 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:41883
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:29.687664 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:29.688783 3014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:29.697265 3019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:29.698118 3020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:29.698920 3022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:29.699769 3014 server_base.cc:1034] running on GCE node
I20250114 03:03:29.700173 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:29.700732 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:29.701890 3014 hybrid_clock.cc:648] HybridClock initialized: now 1736823809701876 us; error 25 us; skew 500 ppm
I20250114 03:03:29.703811 3014 webserver.cc:458] Webserver started at http://127.1.109.65:39085/ using document root <none> and password file <none>
I20250114 03:03:29.704468 3014 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:29.704557 3014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:29.704764 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:29.707022 3014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2d76dc37a682421084593a4712c3be9e"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.707667 3014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2d76dc37a682421084593a4712c3be9e"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.711689 3014 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20250114 03:03:29.713795 3028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.714900 3014 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:03:29.715013 3014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2d76dc37a682421084593a4712c3be9e"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.715107 3014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:29.745684 3014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:29.746345 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:29.746523 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:29.747080 3014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:29.747979 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:03:29.748030 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.748090 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:03:29.748123 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.756664 3014 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:35251
I20250114 03:03:29.756693 3142 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:35251 every 8 connection(s)
I20250114 03:03:29.757666 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:03:29.766279 3143 heartbeater.cc:346] Connected to a master server at 127.1.109.126:41883
I20250114 03:03:29.766489 3143 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:29.766991 3143 heartbeater.cc:510] Master 127.1.109.126:41883 requested a full tablet report, sending...
I20250114 03:03:29.767251 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 3014
I20250114 03:03:29.767355 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:03:29.768537 2954 ts_manager.cc:194] Registered new tserver with Master: 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:29.769662 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:41883
--builtin_ntp_servers=127.1.109.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250114 03:03:29.769837 2954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:50191
W20250114 03:03:29.890082 3146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:29.890314 3146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:29.890364 3146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:29.893554 3146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:29.893690 3146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:03:29.897475 3146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:41883
--never_fsync=true
--heap_profile_path=/tmp/kudu.3146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:29.898540 3146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:29.899689 3146 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:29.908586 3152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:29.909260 3151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:29.910296 3154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:29.911211 3146 server_base.cc:1034] running on GCE node
I20250114 03:03:29.911613 3146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:29.912386 3146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:29.913563 3146 hybrid_clock.cc:648] HybridClock initialized: now 1736823809913542 us; error 27 us; skew 500 ppm
I20250114 03:03:29.915526 3146 webserver.cc:458] Webserver started at http://127.1.109.66:33315/ using document root <none> and password file <none>
I20250114 03:03:29.916134 3146 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:29.916188 3146 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:29.916368 3146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:29.918183 3146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0b7594b0f72b47b796c0f3b8c608811f"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.918663 3146 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0b7594b0f72b47b796c0f3b8c608811f"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.922256 3146 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:03:29.924556 3160 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.925652 3146 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:03:29.925768 3146 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0b7594b0f72b47b796c0f3b8c608811f"
format_stamp: "Formatted at 2025-01-14 03:03:29 on dist-test-slave-r16g"
I20250114 03:03:29.925868 3146 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:29.942107 3146 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:29.942745 3146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:29.942926 3146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:29.943455 3146 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:29.944376 3146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:03:29.944424 3146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.944482 3146 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:03:29.944514 3146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:29.953310 3146 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:35013
I20250114 03:03:29.953341 3274 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:35013 every 8 connection(s)
I20250114 03:03:29.954212 3146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:03:29.955513 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 3146
I20250114 03:03:29.955667 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:03:29.957980 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:41883
--builtin_ntp_servers=127.1.109.84:40317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250114 03:03:29.963805 3275 heartbeater.cc:346] Connected to a master server at 127.1.109.126:41883
I20250114 03:03:29.964082 3275 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:29.964920 3275 heartbeater.cc:510] Master 127.1.109.126:41883 requested a full tablet report, sending...
I20250114 03:03:29.966029 2954 ts_manager.cc:194] Registered new tserver with Master: 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013)
I20250114 03:03:29.966562 2954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:33063
W20250114 03:03:30.065277 3278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:03:30.065524 3278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:03:30.065580 3278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:03:30.068720 3278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:03:30.068861 3278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:03:30.072479 3278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:40317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:41883
--never_fsync=true
--heap_profile_path=/tmp/kudu.3278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:03:30.073616 3278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:03:30.074787 3278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:03:30.084110 3286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:03:30.084920 3283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:30.085732 3278 server_base.cc:1034] running on GCE node
W20250114 03:03:30.086709 3284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:03:30.087177 3278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:03:30.087792 3278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:03:30.088944 3278 hybrid_clock.cc:648] HybridClock initialized: now 1736823810088917 us; error 39 us; skew 500 ppm
I20250114 03:03:30.090973 3278 webserver.cc:458] Webserver started at http://127.1.109.67:42577/ using document root <none> and password file <none>
I20250114 03:03:30.091607 3278 fs_manager.cc:362] Metadata directory not provided
I20250114 03:03:30.091670 3278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:03:30.091867 3278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:03:30.093645 3278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5a9be6271ae946edad7bd7193ae7419f"
format_stamp: "Formatted at 2025-01-14 03:03:30 on dist-test-slave-r16g"
I20250114 03:03:30.094098 3278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5a9be6271ae946edad7bd7193ae7419f"
format_stamp: "Formatted at 2025-01-14 03:03:30 on dist-test-slave-r16g"
I20250114 03:03:30.097481 3278 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250114 03:03:30.099558 3292 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:30.100502 3278 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:03:30.100611 3278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5a9be6271ae946edad7bd7193ae7419f"
format_stamp: "Formatted at 2025-01-14 03:03:30 on dist-test-slave-r16g"
I20250114 03:03:30.100689 3278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:03:30.111001 3278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:03:30.111552 3278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:03:30.111706 3278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:03:30.112205 3278 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:03:30.113152 3278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:03:30.113202 3278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:30.113261 3278 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:03:30.113294 3278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:03:30.121472 3278 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:37995
I20250114 03:03:30.121528 3406 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:37995 every 8 connection(s)
I20250114 03:03:30.122421 3278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:03:30.123672 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 3278
I20250114 03:03:30.123770 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:03:30.131191 3407 heartbeater.cc:346] Connected to a master server at 127.1.109.126:41883
I20250114 03:03:30.131378 3407 heartbeater.cc:463] Registering TS with master...
I20250114 03:03:30.131848 3407 heartbeater.cc:510] Master 127.1.109.126:41883 requested a full tablet report, sending...
I20250114 03:03:30.132763 2954 ts_manager.cc:194] Registered new tserver with Master: 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995)
I20250114 03:03:30.133324 2954 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:39445
I20250114 03:03:30.137656 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:03:30.139034 1461 test_util.cc:274] Using random seed: -834821228
I20250114 03:03:30.148803 2954 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49418:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 03:03:30.149914 2954 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:03:30.165824 3208 tablet_service.cc:1467] Processing CreateTablet for tablet 758ade1cfcb14ffe9b6d10cff41b76df (DEFAULT_TABLE table=test-workload [id=1e2af0673b2d4d87ad600234299ba054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:03:30.166369 3076 tablet_service.cc:1467] Processing CreateTablet for tablet 758ade1cfcb14ffe9b6d10cff41b76df (DEFAULT_TABLE table=test-workload [id=1e2af0673b2d4d87ad600234299ba054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:03:30.166857 3208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758ade1cfcb14ffe9b6d10cff41b76df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:30.167165 3076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758ade1cfcb14ffe9b6d10cff41b76df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:30.167850 3340 tablet_service.cc:1467] Processing CreateTablet for tablet 758ade1cfcb14ffe9b6d10cff41b76df (DEFAULT_TABLE table=test-workload [id=1e2af0673b2d4d87ad600234299ba054]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:03:30.168691 3340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758ade1cfcb14ffe9b6d10cff41b76df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:03:30.172123 3424 tablet_bootstrap.cc:492] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e: Bootstrap starting.
I20250114 03:03:30.173368 3425 tablet_bootstrap.cc:492] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: Bootstrap starting.
I20250114 03:03:30.174710 3424 tablet_bootstrap.cc:654] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:30.174852 3426 tablet_bootstrap.cc:492] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f: Bootstrap starting.
I20250114 03:03:30.175449 3425 tablet_bootstrap.cc:654] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:30.175671 3424 log.cc:826] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:30.176117 3425 log.cc:826] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:30.176851 3426 tablet_bootstrap.cc:654] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f: Neither blocks nor log segments found. Creating new log.
I20250114 03:03:30.177546 3424 tablet_bootstrap.cc:492] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e: No bootstrap required, opened a new log
I20250114 03:03:30.177569 3425 tablet_bootstrap.cc:492] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: No bootstrap required, opened a new log
I20250114 03:03:30.177585 3426 log.cc:826] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f: Log is configured to *not* fsync() on all Append() calls
I20250114 03:03:30.177733 3425 ts_tablet_manager.cc:1397] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:03:30.177733 3424 ts_tablet_manager.cc:1397] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250114 03:03:30.179200 3426 tablet_bootstrap.cc:492] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f: No bootstrap required, opened a new log
I20250114 03:03:30.179358 3426 ts_tablet_manager.cc:1397] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:03:30.180939 3425 raft_consensus.cc:357] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.181125 3425 raft_consensus.cc:383] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:30.181180 3425 raft_consensus.cc:738] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7594b0f72b47b796c0f3b8c608811f, State: Initialized, Role: FOLLOWER
I20250114 03:03:30.181582 3425 consensus_queue.cc:260] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.182204 3275 heartbeater.cc:502] Master 127.1.109.126:41883 was elected leader, sending a full tablet report...
I20250114 03:03:30.182163 3424 raft_consensus.cc:357] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.182346 3424 raft_consensus.cc:383] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:30.182394 3424 raft_consensus.cc:738] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d76dc37a682421084593a4712c3be9e, State: Initialized, Role: FOLLOWER
I20250114 03:03:30.182451 3425 ts_tablet_manager.cc:1428] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:03:30.182705 3430 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:03:30.182828 3430 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.182853 3424 consensus_queue.cc:260] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.183075 3426 raft_consensus.cc:357] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.183264 3426 raft_consensus.cc:383] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:03:30.183308 3426 raft_consensus.cc:738] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a9be6271ae946edad7bd7193ae7419f, State: Initialized, Role: FOLLOWER
I20250114 03:03:30.183676 3426 consensus_queue.cc:260] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.184314 3433 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:03:30.184434 3426 ts_tablet_manager.cc:1428] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250114 03:03:30.184486 3407 heartbeater.cc:502] Master 127.1.109.126:41883 was elected leader, sending a full tablet report...
I20250114 03:03:30.184581 3433 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.185636 3433 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995)
I20250114 03:03:30.185773 3434 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:03:30.185884 3434 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.186188 3430 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:30.186854 3143 heartbeater.cc:502] Master 127.1.109.126:41883 was elected leader, sending a full tablet report...
I20250114 03:03:30.190701 3434 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:30.190963 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:30.191148 3229 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 0.
I20250114 03:03:30.191627 3032 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e; no voters:
I20250114 03:03:30.191912 3431 raft_consensus.cc:2798] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:03:30.191984 3431 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:03:30.192039 3431 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 0 FOLLOWER]: Advancing to term 1
W20250114 03:03:30.193019 3164 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.67:37995 timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20250114 03:03:30.193274 3431 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.193619 3431 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 1 election: Requested vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995)
W20250114 03:03:30.193919 3296 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.66:35013 timed out after 0.006s (SENT)
W20250114 03:03:30.194000 3032 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.67:37995 timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20250114 03:03:30.194518 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f"
I20250114 03:03:30.194638 3228 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:30.194666 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f"
I20250114 03:03:30.194809 3361 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:03:30.195689 3228 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 1.
I20250114 03:03:30.195712 3424 ts_tablet_manager.cc:1428] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 03:03:30.196008 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:30.196148 3229 raft_consensus.cc:2388] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a9be6271ae946edad7bd7193ae7419f in current term 1: Already voted for candidate 2d76dc37a682421084593a4712c3be9e in this term.
I20250114 03:03:30.196180 3361 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 1.
I20250114 03:03:30.196555 3032 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f; no voters:
I20250114 03:03:30.196946 3431 raft_consensus.cc:2798] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:03:30.197041 3431 raft_consensus.cc:695] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 1 LEADER]: Becoming Leader. State: Replica: 2d76dc37a682421084593a4712c3be9e, State: Running, Role: LEADER
I20250114 03:03:30.197336 3431 consensus_queue.cc:237] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:30.198518 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:30.198694 3361 raft_consensus.cc:2388] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0b7594b0f72b47b796c0f3b8c608811f in current term 1: Already voted for candidate 2d76dc37a682421084593a4712c3be9e in this term.
I20250114 03:03:30.199128 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:30.199213 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:30.199219 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:30.199296 3361 raft_consensus.cc:2371] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 1. Re-sending same reply.
I20250114 03:03:30.199254 3229 raft_consensus.cc:2388] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a9be6271ae946edad7bd7193ae7419f in current term 1: Already voted for candidate 2d76dc37a682421084593a4712c3be9e in this term.
I20250114 03:03:30.199326 3095 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:30.199831 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:30.199960 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:30.200317 3430 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 03:03:30.200464 3434 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 03:03:30.202072 3431 consensus_queue.cc:1035] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:03:30.202176 2954 catalog_manager.cc:5526] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d76dc37a682421084593a4712c3be9e (127.1.109.65). New cstate: current_term: 1 leader_uuid: "2d76dc37a682421084593a4712c3be9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } health_report { overall_health: HEALTHY } } }
W20250114 03:03:30.205252 3276 tablet.cc:2367] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:03:30.225951 3448 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:03:30.226028 3448 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250114 03:03:31.010464 3361 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:03:31.010470 3229 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:03:31.010533 3361 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250114 03:03:31.010536 3229 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250114 03:03:31.202080 3457 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:31.202214 3457 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:31.202991 3457 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:31.203428 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:31.203446 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:31.203913 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:31.204094 3457 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:03:31.209386 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.843s user 0.000s sys 0.000s
W20250114 03:03:31.209736 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.843s user 0.001s sys 0.000s
I20250114 03:03:31.303274 3456 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:31.303416 3456 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:31.304117 3456 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:31.304766 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:31.304956 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:31.305430 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:31.305696 3456 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:31.998414 3456 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:31.998567 3456 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:31.999106 3456 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:31.999523 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:31.999509 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:32.000074 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:32.001451 3456 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:32.001528 3467 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=1, attempt_no=0}) took 1779 ms. Trace:
W20250114 03:03:32.001933 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.642s user 0.000s sys 0.000s
W20250114 03:03:32.001940 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.642s user 0.000s sys 0.000s
I20250114 03:03:32.001614 3467 rpcz_store.cc:276] 0114 03:03:30.222105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:30.222152 (+ 47us) service_pool.cc:225] Handling call
0114 03:03:32.001510 (+1779358us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:31.008979 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:31.009097 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:31.009102 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:31.009104 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:31.009178 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:31.009203 (+ 25us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:31.009207 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:31.009250 (+ 43us) tablet.cc:698] Row locks acquired
0114 03:03:31.009252 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:31.009294 (+ 42us) write_op.cc:270] Start()
0114 03:03:31.739423 (+730129us) write_op.cc:276] Timestamp: P: 1736823811009238 usec, L: 0
0114 03:03:31.739427 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:31.739651 (+ 224us) log.cc:844] Serialized 345 byte log entry
0114 03:03:31.999389 (+259738us) op_driver.cc:464] REPLICATION: finished
0114 03:03:31.999644 (+ 255us) write_op.cc:301] APPLY: starting
0114 03:03:31.999665 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:31.999700 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:31.999715 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:31.999785 (+ 70us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:31.999814 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:31.999818 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:32.000084 (+ 266us) log.cc:844] Serialized 22 byte log entry
0114 03:03:32.001408 (+ 1324us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:32.001424 (+ 16us) write_op.cc:454] Released schema lock
0114 03:03:32.001429 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":904448,"child_traces":[["op",{"apply.queue_time_us":179,"num_ops":1,"prepare.queue_time_us":784412,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":730763,"raft.queue_time_us":40,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":132,"replication_time_us":259927,"thread_start_us":93,"threads_started":1}]]}
I20250114 03:03:32.118491 3465 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:32.118638 3465 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:32.119163 3465 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:32.119531 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:32.119736 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:32.120167 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:32.120386 3465 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:33.029104 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:33.029258 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:33.029801 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:33.030230 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:33.030258 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:33.030772 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:33.030929 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:33.031988 3475 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=2, attempt_no=0}) took 1825 ms. Trace:
I20250114 03:03:33.032073 3475 rpcz_store.cc:276] 0114 03:03:31.206950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:31.207010 (+ 60us) service_pool.cc:225] Handling call
0114 03:03:33.031973 (+1824963us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:31.739773 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:31.739887 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:31.739892 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:31.739894 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:31.739970 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:31.739978 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:31.739981 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:31.740051 (+ 70us) tablet.cc:698] Row locks acquired
0114 03:03:31.740054 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:03:31.740098 (+ 44us) write_op.cc:270] Start()
0114 03:03:32.687259 (+947161us) write_op.cc:276] Timestamp: P: 1736823811740022 usec, L: 0
0114 03:03:32.687264 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:03:32.687530 (+ 266us) log.cc:844] Serialized 354 byte log entry
0114 03:03:33.030196 (+342666us) op_driver.cc:464] REPLICATION: finished
0114 03:03:33.030935 (+ 739us) write_op.cc:301] APPLY: starting
0114 03:03:33.030956 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:33.030991 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:33.031005 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:33.031066 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:33.031100 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:33.031106 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:03:33.031175 (+ 69us) log.cc:844] Serialized 22 byte log entry
0114 03:03:33.031864 (+ 689us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:33.031879 (+ 15us) write_op.cc:454] Released schema lock
0114 03:03:33.031884 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":264,"num_ops":1,"prepare.queue_time_us":532298,"prepare.run_cpu_time_us":792,"prepare.run_wall_time_us":947849,"raft.queue_time_us":158,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":342897,"thread_start_us":67,"threads_started":1}]]}
W20250114 03:03:33.033228 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.881s user 0.001s sys 0.000s
W20250114 03:03:33.033471 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.881s user 0.001s sys 0.000s
I20250114 03:03:33.554051 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:33.554226 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:33.554822 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:33.555222 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:33.555747 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:33.556160 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:33.592406 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:33.592592 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:33.593127 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:33.593549 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:33.593537 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:33.594053 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:33.594429 3482 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=3, attempt_no=0}) took 1590 ms. Trace:
I20250114 03:03:33.594503 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:33.594548 3482 rpcz_store.cc:276] 0114 03:03:32.003521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:32.003578 (+ 57us) service_pool.cc:225] Handling call
0114 03:03:33.594398 (+1590820us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:32.687672 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:32.687808 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:32.687815 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:03:32.687819 (+ 4us) tablet.cc:659] Decoding operations
0114 03:03:32.687909 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:32.687919 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:32.687923 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:32.687971 (+ 48us) tablet.cc:698] Row locks acquired
0114 03:03:32.687972 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:32.688015 (+ 43us) write_op.cc:270] Start()
0114 03:03:33.394155 (+706140us) write_op.cc:276] Timestamp: P: 1736823812687951 usec, L: 0
0114 03:03:33.394158 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:03:33.394404 (+ 246us) log.cc:844] Serialized 354 byte log entry
0114 03:03:33.593651 (+199247us) op_driver.cc:464] REPLICATION: finished
0114 03:03:33.593910 (+ 259us) write_op.cc:301] APPLY: starting
0114 03:03:33.593928 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:33.593959 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:33.593975 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:33.594032 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:33.594060 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:33.594066 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:03:33.594132 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:03:33.594213 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:33.594226 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:33.594232 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":204,"num_ops":1,"prepare.queue_time_us":683784,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":706828,"replication_time_us":199274,"thread_start_us":72,"threads_started":1}]]}
I20250114 03:03:33.646600 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:33.710048 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:33.710191 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:33.710682 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:33.711227 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:33.711716 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:33.712109 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:33.712294 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:34.248528 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:34.248687 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:34.249220 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:34.249639 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:34.249644 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:34.250190 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:34.250327 3488 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=4, attempt_no=0}) took 1216 ms. Trace:
I20250114 03:03:34.250391 3488 rpcz_store.cc:276] 0114 03:03:33.033426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:33.033479 (+ 53us) service_pool.cc:225] Handling call
0114 03:03:34.250315 (+1216836us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:33.394540 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:33.394688 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:33.394693 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:33.394694 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:33.394773 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:33.394781 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:33.394785 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:33.394829 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:03:33.394830 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:33.394863 (+ 33us) write_op.cc:270] Start()
0114 03:03:33.741004 (+346141us) write_op.cc:276] Timestamp: P: 1736823813394827 usec, L: 0
0114 03:03:33.741008 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:33.741188 (+ 180us) log.cc:844] Serialized 354 byte log entry
0114 03:03:34.249688 (+508500us) op_driver.cc:464] REPLICATION: finished
0114 03:03:34.249905 (+ 217us) write_op.cc:301] APPLY: starting
0114 03:03:34.249928 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:34.249960 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:34.249973 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:34.250040 (+ 67us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:34.250070 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:34.250076 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:03:34.250140 (+ 64us) log.cc:844] Serialized 22 byte log entry
0114 03:03:34.250226 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:34.250238 (+ 12us) write_op.cc:454] Released schema lock
0114 03:03:34.250243 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":173,"num_ops":1,"prepare.queue_time_us":360780,"prepare.run_cpu_time_us":918,"prepare.run_wall_time_us":346722,"replication_time_us":508646,"thread_start_us":77,"threads_started":1}]]}
I20250114 03:03:34.302093 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:03:34.304728 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.559s user 0.001s sys 0.000s
W20250114 03:03:34.304759 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.559s user 0.001s sys 0.000s
I20250114 03:03:34.315220 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:34.315336 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:34.315817 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:34.316201 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:34.316392 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:34.316994 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:35.069257 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:35.162007 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:35.162142 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:35.162647 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:35.163163 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:35.163211 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:35.164326 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:35.164582 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:03:35.164855 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.710s user 0.001s sys 0.000s
W20250114 03:03:35.164855 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.710s user 0.001s sys 0.000s
I20250114 03:03:35.329856 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:35.330013 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:35.330581 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:35.331030 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:35.331184 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:35.331673 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:35.331743 3496 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=7, attempt_no=0}) took 1027 ms. Trace:
I20250114 03:03:35.331928 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:35.331838 3496 rpcz_store.cc:276] 0114 03:03:34.304655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:34.304724 (+ 69us) service_pool.cc:225] Handling call
0114 03:03:35.331727 (+1027003us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:34.763598 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:34.763693 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:34.763697 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:34.763698 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:34.763758 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:34.763768 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:34.763771 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:34.763813 (+ 42us) tablet.cc:698] Row locks acquired
0114 03:03:34.763814 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:34.763858 (+ 44us) write_op.cc:270] Start()
0114 03:03:34.945041 (+181183us) write_op.cc:276] Timestamp: P: 1736823814763827 usec, L: 0
0114 03:03:34.945045 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:34.945255 (+ 210us) log.cc:844] Serialized 354 byte log entry
0114 03:03:35.331042 (+385787us) op_driver.cc:464] REPLICATION: finished
0114 03:03:35.331156 (+ 114us) write_op.cc:301] APPLY: starting
0114 03:03:35.331176 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:35.331209 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:35.331223 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:35.331277 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:35.331310 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:35.331319 (+ 9us) write_op.cc:312] APPLY: finished
0114 03:03:35.331386 (+ 67us) log.cc:844] Serialized 22 byte log entry
0114 03:03:35.331626 (+ 240us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:35.331641 (+ 15us) write_op.cc:454] Released schema lock
0114 03:03:35.331649 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":1,"prepare.queue_time_us":458529,"prepare.run_cpu_time_us":678,"prepare.run_wall_time_us":181730,"replication_time_us":385966}]]}
I20250114 03:03:35.927050 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:35.927191 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:35.927699 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:35.928195 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:35.929302 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:35.929731 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:36.045099 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:36.045233 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:36.045912 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:36.046299 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:36.046350 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:36.046780 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
W20250114 03:03:36.048744 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.733s user 0.001s sys 0.000s
W20250114 03:03:36.048769 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.733s user 0.001s sys 0.000s
I20250114 03:03:36.840402 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:36.842231 3510 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=9, attempt_no=0}) took 1509 ms. Trace:
I20250114 03:03:36.842321 3510 rpcz_store.cc:276] 0114 03:03:35.333182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:35.333268 (+ 86us) service_pool.cc:225] Handling call
0114 03:03:36.842214 (+1508946us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:35.333652 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:35.333748 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:35.333751 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:35.333752 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:35.333807 (+ 55us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:35.333816 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:35.333821 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:35.333871 (+ 50us) tablet.cc:698] Row locks acquired
0114 03:03:35.333874 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:03:35.333905 (+ 31us) write_op.cc:270] Start()
0114 03:03:35.434092 (+100187us) write_op.cc:276] Timestamp: P: 1736823815333875 usec, L: 0
0114 03:03:35.434096 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:35.434318 (+ 222us) log.cc:844] Serialized 354 byte log entry
0114 03:03:36.841497 (+1407179us) op_driver.cc:464] REPLICATION: finished
0114 03:03:36.841830 (+ 333us) write_op.cc:301] APPLY: starting
0114 03:03:36.841848 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:36.841878 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:36.841894 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:36.841963 (+ 69us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:36.841987 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:36.841991 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:36.842049 (+ 58us) log.cc:844] Serialized 22 byte log entry
0114 03:03:36.842121 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:36.842135 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:36.842140 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":273,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":647,"prepare.run_wall_time_us":100725,"replication_time_us":1407372,"thread_start_us":128,"threads_started":1}]]}
I20250114 03:03:37.017060 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:37.477546 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:37.477685 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:37.478346 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:37.478863 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:37.478874 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:37.480183 3516 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=10, attempt_no=0}) took 1431 ms. Trace:
I20250114 03:03:37.480294 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 26) took 1280 ms. Trace:
I20250114 03:03:37.480389 3055 rpcz_store.cc:276] 0114 03:03:36.200057 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:36.200122 (+ 65us) service_pool.cc:225] Handling call
0114 03:03:36.200354 (+ 232us) tablet_service.cc:2780] Created scanner 3a9c56170ae2489c9ae14fffe067a4d9 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 0f2766d261b44640b3473b1b57999394
0114 03:03:36.200581 (+ 227us) tablet_service.cc:2920] Creating iterator
0114 03:03:36.200635 (+ 54us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:36.656788 (+456153us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:37.479407 (+822619us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1278763 microseconds
0114 03:03:37.479490 (+ 83us) tablet_service.cc:2945] Iterator created
0114 03:03:37.479785 (+ 295us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:37.479811 (+ 26us) tablet_service.cc:3010] has_more: true
0114 03:03:37.479846 (+ 35us) tablet_service.cc:3027] Continuing scan request
0114 03:03:37.479924 (+ 78us) tablet_service.cc:3091] Found scanner 3a9c56170ae2489c9ae14fffe067a4d9 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 0f2766d261b44640b3473b1b57999394
0114 03:03:37.480288 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":84480}
I20250114 03:03:37.480481 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:37.480294 3516 rpcz_store.cc:276] 0114 03:03:36.048686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:36.048726 (+ 40us) service_pool.cc:225] Handling call
0114 03:03:37.480169 (+1431443us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:36.049299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:36.049366 (+ 67us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:36.049369 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:36.049370 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:36.049418 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:36.049423 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:36.049425 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:36.049453 (+ 28us) tablet.cc:698] Row locks acquired
0114 03:03:36.049454 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:36.049478 (+ 24us) write_op.cc:270] Start()
0114 03:03:36.656621 (+607143us) write_op.cc:276] Timestamp: P: 1736823816049424 usec, L: 0
0114 03:03:36.656625 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:36.656919 (+ 294us) log.cc:844] Serialized 354 byte log entry
0114 03:03:37.478649 (+821730us) op_driver.cc:464] REPLICATION: finished
0114 03:03:37.479060 (+ 411us) write_op.cc:301] APPLY: starting
0114 03:03:37.479079 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:37.479110 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:37.479126 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:37.479200 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:37.479228 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:37.479233 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:37.479306 (+ 73us) log.cc:844] Serialized 22 byte log entry
0114 03:03:37.480084 (+ 778us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:37.480095 (+ 11us) write_op.cc:454] Released schema lock
0114 03:03:37.480101 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":356,"num_ops":1,"prepare.queue_time_us":168,"prepare.run_cpu_time_us":621,"prepare.run_wall_time_us":607676,"replication_time_us":821987,"thread_start_us":160,"threads_started":2}]]}
W20250114 03:03:37.480927 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.282s user 0.001s sys 0.000s
I20250114 03:03:37.480901 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 25) took 1280 ms. Trace:
I20250114 03:03:37.481079 3056 rpcz_store.cc:276] 0114 03:03:36.199989 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:36.200090 (+ 101us) service_pool.cc:225] Handling call
0114 03:03:36.200325 (+ 235us) tablet_service.cc:2780] Created scanner 2af96d099fb244b390d0fae14b6d7a94 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 867e51d8d06e4fa29a58fb23f21556ae
0114 03:03:36.200585 (+ 260us) tablet_service.cc:2920] Creating iterator
0114 03:03:36.200638 (+ 53us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:36.656788 (+456150us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:37.480109 (+823321us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1279455 microseconds
0114 03:03:37.480198 (+ 89us) tablet_service.cc:2945] Iterator created
0114 03:03:37.480459 (+ 261us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:37.480482 (+ 23us) tablet_service.cc:3010] has_more: true
0114 03:03:37.480517 (+ 35us) tablet_service.cc:3027] Continuing scan request
0114 03:03:37.480554 (+ 37us) tablet_service.cc:3091] Found scanner 2af96d099fb244b390d0fae14b6d7a94 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 867e51d8d06e4fa29a58fb23f21556ae
0114 03:03:37.480894 (+ 340us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":38528}
W20250114 03:03:37.481554 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.282s user 0.001s sys 0.000s
I20250114 03:03:37.685321 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:37.685487 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:37.685966 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:37.686393 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:37.686393 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:37.686800 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:38.353228 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:38.354875 3522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=11, attempt_no=0}) took 1511 ms. Trace:
I20250114 03:03:38.354996 3522 rpcz_store.cc:276] 0114 03:03:36.843814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:36.843880 (+ 66us) service_pool.cc:225] Handling call
0114 03:03:38.354844 (+1510964us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:36.844322 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:36.844409 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:36.844413 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:36.844416 (+ 3us) tablet.cc:659] Decoding operations
0114 03:03:36.844479 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:36.844486 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:36.844488 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:36.844519 (+ 31us) tablet.cc:698] Row locks acquired
0114 03:03:36.844520 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:36.844556 (+ 36us) write_op.cc:270] Start()
0114 03:03:37.051713 (+207157us) write_op.cc:276] Timestamp: P: 1736823816844533 usec, L: 0
0114 03:03:37.051717 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:37.051966 (+ 249us) log.cc:844] Serialized 354 byte log entry
0114 03:03:38.354314 (+1302348us) op_driver.cc:464] REPLICATION: finished
0114 03:03:38.354501 (+ 187us) write_op.cc:301] APPLY: starting
0114 03:03:38.354517 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:38.354543 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:38.354558 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:38.354607 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:38.354629 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:38.354633 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:38.354689 (+ 56us) log.cc:844] Serialized 22 byte log entry
0114 03:03:38.354753 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:38.354767 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:38.354772 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":149,"num_ops":1,"prepare.queue_time_us":40,"prepare.run_cpu_time_us":637,"prepare.run_wall_time_us":207706,"replication_time_us":1302569,"thread_start_us":60,"threads_started":1}]]}
I20250114 03:03:38.611892 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:38.612046 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:38.612529 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:38.613027 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:38.613080 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 14 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:38.613523 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:38.614413 3522 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=12, attempt_no=0}) took 1132 ms. Trace:
I20250114 03:03:38.614558 3522 rpcz_store.cc:276] 0114 03:03:37.481983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:37.482032 (+ 49us) service_pool.cc:225] Handling call
0114 03:03:38.614403 (+1132371us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:37.482422 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:37.482488 (+ 66us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:37.482490 (+ 2us) write_op.cc:435] Acquired schema lock
0114 03:03:37.482491 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:37.482538 (+ 47us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:37.482543 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:37.482545 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:37.482571 (+ 26us) tablet.cc:698] Row locks acquired
0114 03:03:37.482572 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:37.482594 (+ 22us) write_op.cc:270] Start()
0114 03:03:37.631792 (+149198us) write_op.cc:276] Timestamp: P: 1736823817482561 usec, L: 0
0114 03:03:37.631802 (+ 10us) op_driver.cc:348] REPLICATION: starting
0114 03:03:37.632045 (+ 243us) log.cc:844] Serialized 354 byte log entry
0114 03:03:38.612899 (+980854us) op_driver.cc:464] REPLICATION: finished
0114 03:03:38.613228 (+ 329us) write_op.cc:301] APPLY: starting
0114 03:03:38.613248 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:38.613277 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:38.613292 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:38.613354 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:38.613387 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:38.613392 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:38.613456 (+ 64us) log.cc:844] Serialized 22 byte log entry
0114 03:03:38.614330 (+ 874us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:38.614339 (+ 9us) write_op.cc:454] Released schema lock
0114 03:03:38.614343 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":269,"num_ops":1,"prepare.queue_time_us":36,"prepare.run_cpu_time_us":619,"prepare.run_wall_time_us":149691,"replication_time_us":981052}]]}
W20250114 03:03:38.614882 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.983s user 0.001s sys 0.000s
W20250114 03:03:38.615514 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.983s user 0.001s sys 0.000s
I20250114 03:03:38.656323 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:38.699790 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:38.699929 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:38.700450 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:38.700951 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:38.701122 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:38.702026 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:38.702183 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:38.835350 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:39.287333 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:39.287468 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:39.287959 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:39.288393 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:39.288398 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:39.290026 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
W20250114 03:03:39.290422 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.525s user 0.001s sys 0.000s
W20250114 03:03:39.290526 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.525s user 0.000s sys 0.000s
I20250114 03:03:39.354607 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:39.354743 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:39.355201 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:39.355659 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:39.355669 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 16 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:39.356071 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:40.011577 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:40.013275 3536 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=15, attempt_no=0}) took 1310 ms. Trace:
I20250114 03:03:40.013353 3536 rpcz_store.cc:276] 0114 03:03:38.702496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:38.702560 (+ 64us) service_pool.cc:225] Handling call
0114 03:03:40.013261 (+1310701us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:38.828037 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:38.828157 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:38.828162 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:38.828163 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:38.828245 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:38.828257 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:38.828260 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:38.828306 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:03:38.828308 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:38.828347 (+ 39us) write_op.cc:270] Start()
0114 03:03:39.183535 (+355188us) write_op.cc:276] Timestamp: P: 1736823818828320 usec, L: 0
0114 03:03:39.183539 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:39.183764 (+ 225us) log.cc:844] Serialized 354 byte log entry
0114 03:03:40.012707 (+828943us) op_driver.cc:464] REPLICATION: finished
0114 03:03:40.012914 (+ 207us) write_op.cc:301] APPLY: starting
0114 03:03:40.012934 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:40.012965 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:40.012979 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:40.013038 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:40.013073 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:40.013078 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:40.013137 (+ 59us) log.cc:844] Serialized 22 byte log entry
0114 03:03:40.013187 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:40.013199 (+ 12us) write_op.cc:454] Released schema lock
0114 03:03:40.013203 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":145,"num_ops":1,"prepare.queue_time_us":125147,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":355789,"replication_time_us":829136,"thread_start_us":66,"threads_started":1}]]}
I20250114 03:03:40.019025 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:40.587527 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:40.587669 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:40.588145 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:40.588577 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:40.588691 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:40.589174 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:40.589344 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:40.590271 3543 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=16, attempt_no=0}) took 1300 ms. Trace:
I20250114 03:03:40.590368 3543 rpcz_store.cc:276] 0114 03:03:39.290087 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:39.290177 (+ 90us) service_pool.cc:225] Handling call
0114 03:03:40.590258 (+1300081us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:39.290557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:39.290640 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:39.290643 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:39.290645 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:39.290720 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:39.290726 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:39.290728 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:39.290756 (+ 28us) tablet.cc:698] Row locks acquired
0114 03:03:39.290757 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:39.290783 (+ 26us) write_op.cc:270] Start()
0114 03:03:40.023928 (+733145us) write_op.cc:276] Timestamp: P: 1736823819290709 usec, L: 0
0114 03:03:40.023932 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:40.024169 (+ 237us) log.cc:844] Serialized 354 byte log entry
0114 03:03:40.589624 (+565455us) op_driver.cc:464] REPLICATION: finished
0114 03:03:40.589812 (+ 188us) write_op.cc:301] APPLY: starting
0114 03:03:40.589829 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:40.589859 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:40.589873 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:40.589929 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:40.589955 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:40.589959 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:40.590020 (+ 61us) log.cc:844] Serialized 22 byte log entry
0114 03:03:40.590112 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:40.590126 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:40.590132 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":138,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":623,"prepare.run_wall_time_us":733700,"raft.queue_time_us":220,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":565660,"spinlock_wait_cycles":4864,"thread_start_us":85,"threads_started":1}]]}
I20250114 03:03:40.591125 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 37) took 1149 ms. Trace:
I20250114 03:03:40.591187 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 38) took 1149 ms. Trace:
I20250114 03:03:40.591271 3055 rpcz_store.cc:276] 0114 03:03:39.441711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:39.441785 (+ 74us) service_pool.cc:225] Handling call
0114 03:03:39.441925 (+ 140us) tablet_service.cc:2780] Created scanner 923da134a3d14a6f833e879ebdeb8476 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 28e873f591e944b296c482c6e479d81b
0114 03:03:39.442075 (+ 150us) tablet_service.cc:2920] Creating iterator
0114 03:03:39.442112 (+ 37us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:40.024053 (+581941us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:40.590164 (+566111us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1148034 microseconds
0114 03:03:40.590264 (+ 100us) tablet_service.cc:2945] Iterator created
0114 03:03:40.590583 (+ 319us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:40.590607 (+ 24us) tablet_service.cc:3010] has_more: true
0114 03:03:40.590647 (+ 40us) tablet_service.cc:3027] Continuing scan request
0114 03:03:40.590699 (+ 52us) tablet_service.cc:3091] Found scanner 923da134a3d14a6f833e879ebdeb8476 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 28e873f591e944b296c482c6e479d81b
0114 03:03:40.591177 (+ 478us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":13824}
I20250114 03:03:40.591202 3056 rpcz_store.cc:276] 0114 03:03:39.441635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:39.441697 (+ 62us) service_pool.cc:225] Handling call
0114 03:03:39.441881 (+ 184us) tablet_service.cc:2780] Created scanner e6ade8f5ebe24cbd8bb9c917f9263ff6 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 6e632517196f41a4b0b75104af55e595
0114 03:03:39.442066 (+ 185us) tablet_service.cc:2920] Creating iterator
0114 03:03:39.442113 (+ 47us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:40.024078 (+581965us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:40.590279 (+566201us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1148159 microseconds
0114 03:03:40.590323 (+ 44us) tablet_service.cc:2945] Iterator created
0114 03:03:40.590583 (+ 260us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:40.590611 (+ 28us) tablet_service.cc:3010] has_more: true
0114 03:03:40.590647 (+ 36us) tablet_service.cc:3027] Continuing scan request
0114 03:03:40.590699 (+ 52us) tablet_service.cc:3091] Found scanner e6ade8f5ebe24cbd8bb9c917f9263ff6 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 6e632517196f41a4b0b75104af55e595
0114 03:03:40.591110 (+ 411us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":38016}
W20250114 03:03:40.592116 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.151s user 0.000s sys 0.000s
W20250114 03:03:40.592116 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.151s user 0.000s sys 0.000s
I20250114 03:03:40.786674 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:40.786830 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:40.787330 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:40.787760 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:40.788075 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:40.788434 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:41.227963 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:41.228116 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:41.228581 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:41.228971 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:41.229099 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:41.229532 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:41.229748 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:41.230239 3550 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=17, attempt_no=0}) took 1215 ms. Trace:
I20250114 03:03:41.230314 3550 rpcz_store.cc:276] 0114 03:03:40.014654 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:40.014719 (+ 65us) service_pool.cc:225] Handling call
0114 03:03:41.230223 (+1215504us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:40.024290 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:40.024401 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:40.024405 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:40.024406 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:40.024480 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:40.024488 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:40.024492 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:40.024534 (+ 42us) tablet.cc:698] Row locks acquired
0114 03:03:40.024536 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:40.024577 (+ 41us) write_op.cc:270] Start()
0114 03:03:40.054742 (+ 30165us) write_op.cc:276] Timestamp: P: 1736823820024537 usec, L: 0
0114 03:03:40.054747 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:03:40.055027 (+ 280us) log.cc:844] Serialized 354 byte log entry
0114 03:03:41.229399 (+1174372us) op_driver.cc:464] REPLICATION: finished
0114 03:03:41.229637 (+ 238us) write_op.cc:301] APPLY: starting
0114 03:03:41.229806 (+ 169us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:41.229842 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:41.229856 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:41.229914 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:41.229944 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:41.229948 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:41.230011 (+ 63us) log.cc:844] Serialized 22 byte log entry
0114 03:03:41.230075 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:41.230089 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:41.230094 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":177,"num_ops":1,"prepare.queue_time_us":9217,"prepare.run_cpu_time_us":745,"prepare.run_wall_time_us":30812,"replication_time_us":1174619,"thread_start_us":83,"threads_started":1}]]}
I20250114 03:03:41.606750 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:41.608201 3550 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=18, attempt_no=0}) took 1016 ms. Trace:
I20250114 03:03:41.608350 3550 rpcz_store.cc:276] 0114 03:03:40.592035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:40.592084 (+ 49us) service_pool.cc:225] Handling call
0114 03:03:41.608187 (+1016103us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:40.592541 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:40.592639 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:40.592641 (+ 2us) write_op.cc:435] Acquired schema lock
0114 03:03:40.592642 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:40.592697 (+ 55us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:40.592703 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:40.592705 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:40.592735 (+ 30us) tablet.cc:698] Row locks acquired
0114 03:03:40.592736 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:40.592763 (+ 27us) write_op.cc:270] Start()
0114 03:03:40.636926 (+ 44163us) write_op.cc:276] Timestamp: P: 1736823820592719 usec, L: 0
0114 03:03:40.636930 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:40.637138 (+ 208us) log.cc:844] Serialized 354 byte log entry
0114 03:03:41.607681 (+970543us) op_driver.cc:464] REPLICATION: finished
0114 03:03:41.607768 (+ 87us) write_op.cc:301] APPLY: starting
0114 03:03:41.607786 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:41.607819 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:41.607833 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:41.607889 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:41.607916 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:41.607920 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:41.608004 (+ 84us) log.cc:844] Serialized 22 byte log entry
0114 03:03:41.608096 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:41.608108 (+ 12us) write_op.cc:454] Released schema lock
0114 03:03:41.608113 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":25,"num_ops":1,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":580,"prepare.run_wall_time_us":44657,"replication_time_us":970716,"thread_start_us":74,"threads_started":1}]]}
W20250114 03:03:41.609622 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.867s user 0.001s sys 0.000s
W20250114 03:03:41.609622 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.867s user 0.001s sys 0.000s
I20250114 03:03:41.961930 3471 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:41.962087 3471 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:41.962561 3471 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:41.963088 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:41.963115 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:41.965016 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:41.965206 3471 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:42.035982 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:42.036120 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:42.036598 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:42.037076 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:42.037182 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:42.037658 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:42.405316 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:42.710327 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:42.710486 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:42.710973 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:42.711385 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:42.711416 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:42.711977 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:42.712244 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:42.713400 3560 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=20, attempt_no=0}) took 1103 ms. Trace:
I20250114 03:03:42.713477 3560 rpcz_store.cc:276] 0114 03:03:41.609974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:41.610025 (+ 51us) service_pool.cc:225] Handling call
0114 03:03:42.713386 (+1103361us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:41.820863 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:41.820978 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:41.820985 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:03:41.820988 (+ 3us) tablet.cc:659] Decoding operations
0114 03:03:41.821066 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:41.821075 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:41.821078 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:41.821123 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:03:41.821124 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:41.821160 (+ 36us) write_op.cc:270] Start()
0114 03:03:42.595261 (+774101us) write_op.cc:276] Timestamp: P: 1736823821821094 usec, L: 0
0114 03:03:42.595265 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:42.595493 (+ 228us) log.cc:844] Serialized 354 byte log entry
0114 03:03:42.711251 (+115758us) op_driver.cc:464] REPLICATION: finished
0114 03:03:42.711659 (+ 408us) write_op.cc:301] APPLY: starting
0114 03:03:42.711677 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:42.711934 (+ 257us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:42.711952 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:42.712019 (+ 67us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:42.712053 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:42.712057 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:42.712125 (+ 68us) log.cc:844] Serialized 22 byte log entry
0114 03:03:42.713296 (+ 1171us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:42.713309 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:42.713313 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":343,"num_ops":1,"prepare.queue_time_us":210447,"prepare.run_cpu_time_us":712,"prepare.run_wall_time_us":774738,"raft.queue_time_us":165,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":283,"replication_time_us":115948,"thread_start_us":143,"threads_started":1}]]}
W20250114 03:03:42.713781 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.598s user 0.001s sys 0.000s
W20250114 03:03:42.714385 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.599s user 0.001s sys 0.000s
I20250114 03:03:43.136391 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:43.136521 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:43.137090 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:43.137449 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:43.137584 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:43.137972 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:43.811003 3472 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:43.811168 3472 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:43.811668 3472 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:43.812104 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:43.812110 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 23 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:43.812642 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:43.813030 3568 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=21, attempt_no=0}) took 1847 ms. Trace:
I20250114 03:03:43.813148 3568 rpcz_store.cc:276] 0114 03:03:41.965170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:41.965217 (+ 47us) service_pool.cc:225] Handling call
0114 03:03:43.813014 (+1847797us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:42.595631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:42.595760 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:42.595767 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:03:42.595770 (+ 3us) tablet.cc:659] Decoding operations
0114 03:03:42.595843 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:42.595853 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:42.595855 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:42.595902 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:03:42.595905 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:03:42.595943 (+ 38us) write_op.cc:270] Start()
0114 03:03:43.084111 (+488168us) write_op.cc:276] Timestamp: P: 1736823822595901 usec, L: 0
0114 03:03:43.084116 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:03:43.084373 (+ 257us) log.cc:844] Serialized 354 byte log entry
0114 03:03:43.812342 (+727969us) op_driver.cc:464] REPLICATION: finished
0114 03:03:43.812573 (+ 231us) write_op.cc:301] APPLY: starting
0114 03:03:43.812594 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:43.812628 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:43.812642 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:43.812703 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:43.812734 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:43.812738 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:43.812804 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:03:43.812924 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:43.812938 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:43.812943 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":182,"num_ops":1,"prepare.queue_time_us":630043,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":488811,"raft.queue_time_us":31,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"replication_time_us":728194,"spinlock_wait_cycles":45440,"thread_start_us":66,"threads_started":1}]]}
W20250114 03:03:43.814239 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.949s user 0.000s sys 0.000s
W20250114 03:03:43.814239 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.950s user 0.001s sys 0.000s
I20250114 03:03:44.002768 3472 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:44.004268 3568 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=22, attempt_no=0}) took 1289 ms. Trace:
I20250114 03:03:44.004410 3568 rpcz_store.cc:276] 0114 03:03:42.714811 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:42.714863 (+ 52us) service_pool.cc:225] Handling call
0114 03:03:44.004255 (+1289392us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:43.084475 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:43.084625 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:43.084630 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:43.084631 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:43.084707 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:43.084719 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:43.084723 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:43.084769 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:03:43.084771 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:43.084812 (+ 41us) write_op.cc:270] Start()
0114 03:03:43.518013 (+433201us) write_op.cc:276] Timestamp: P: 1736823823084774 usec, L: 0
0114 03:03:43.518017 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:43.518235 (+ 218us) log.cc:844] Serialized 354 byte log entry
0114 03:03:44.003671 (+485436us) op_driver.cc:464] REPLICATION: finished
0114 03:03:44.003758 (+ 87us) write_op.cc:301] APPLY: starting
0114 03:03:44.003774 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:44.003805 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:44.003822 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:44.003884 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:44.003910 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:44.003915 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:44.003977 (+ 62us) log.cc:844] Serialized 22 byte log entry
0114 03:03:44.004148 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:44.004160 (+ 12us) write_op.cc:454] Released schema lock
0114 03:03:44.004165 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":50,"num_ops":1,"prepare.queue_time_us":369290,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":433824,"replication_time_us":485621}]]}
I20250114 03:03:44.098932 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:44.767002 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:44.767153 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:44.767627 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:44.768131 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:44.768333 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:44.768676 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:45.117223 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:45.117360 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:45.117877 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:45.118307 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:45.118317 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:45.118737 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:45.119035 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:45.119189 3578 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=23, attempt_no=0}) took 1304 ms. Trace:
I20250114 03:03:45.119247 3578 rpcz_store.cc:276] 0114 03:03:43.814679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:43.814733 (+ 54us) service_pool.cc:225] Handling call
0114 03:03:45.119175 (+1304442us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:43.815060 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:43.815129 (+ 69us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:43.815133 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:43.815134 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:43.815182 (+ 48us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:43.815188 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:43.815190 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:43.815224 (+ 34us) tablet.cc:698] Row locks acquired
0114 03:03:43.815225 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:43.815250 (+ 25us) write_op.cc:270] Start()
0114 03:03:44.632393 (+817143us) write_op.cc:276] Timestamp: P: 1736823823815195 usec, L: 0
0114 03:03:44.632397 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:44.632667 (+ 270us) log.cc:844] Serialized 354 byte log entry
0114 03:03:45.118411 (+485744us) op_driver.cc:464] REPLICATION: finished
0114 03:03:45.118763 (+ 352us) write_op.cc:301] APPLY: starting
0114 03:03:45.118779 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:45.118808 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:45.118822 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:45.118908 (+ 86us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:45.118935 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:45.118940 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:45.119007 (+ 67us) log.cc:844] Serialized 22 byte log entry
0114 03:03:45.119084 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:45.119097 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:45.119102 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":298,"num_ops":1,"prepare.queue_time_us":37,"prepare.run_cpu_time_us":618,"prepare.run_wall_time_us":817688,"raft.queue_time_us":40,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":180,"replication_time_us":485982,"spinlock_wait_cycles":74112,"thread_start_us":66,"threads_started":1}]]}
I20250114 03:03:45.120139 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 55) took 1154 ms. Trace:
I20250114 03:03:45.120211 3056 rpcz_store.cc:276] 0114 03:03:43.965625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:43.965688 (+ 63us) service_pool.cc:225] Handling call
0114 03:03:43.965910 (+ 222us) tablet_service.cc:2780] Created scanner c755195244584ad59d199ffedd294c21 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 282b1ba3708c4eba9c96ee3d9df06c0b
0114 03:03:43.966085 (+ 175us) tablet_service.cc:2920] Creating iterator
0114 03:03:43.966127 (+ 42us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:44.632538 (+666411us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:45.119157 (+486619us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1153020 microseconds
0114 03:03:45.119241 (+ 84us) tablet_service.cc:2945] Iterator created
0114 03:03:45.119567 (+ 326us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:45.119613 (+ 46us) tablet_service.cc:3010] has_more: true
0114 03:03:45.119647 (+ 34us) tablet_service.cc:3027] Continuing scan request
0114 03:03:45.119714 (+ 67us) tablet_service.cc:3091] Found scanner c755195244584ad59d199ffedd294c21 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 282b1ba3708c4eba9c96ee3d9df06c0b
0114 03:03:45.120129 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":62208}
I20250114 03:03:45.120157 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 54) took 1154 ms. Trace:
I20250114 03:03:45.120353 3055 rpcz_store.cc:276] 0114 03:03:43.965546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:43.965658 (+ 112us) service_pool.cc:225] Handling call
0114 03:03:43.965895 (+ 237us) tablet_service.cc:2780] Created scanner 59a9d7684ef74d7782385fb753644bbe for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is aff82a89f7894c9d8597ace5f998b5be
0114 03:03:43.966085 (+ 190us) tablet_service.cc:2920] Creating iterator
0114 03:03:43.966129 (+ 44us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:44.632552 (+666423us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:45.119186 (+486634us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1153043 microseconds
0114 03:03:45.119265 (+ 79us) tablet_service.cc:2945] Iterator created
0114 03:03:45.119564 (+ 299us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:45.119588 (+ 24us) tablet_service.cc:3010] has_more: true
0114 03:03:45.119649 (+ 61us) tablet_service.cc:3027] Continuing scan request
0114 03:03:45.119705 (+ 56us) tablet_service.cc:3091] Found scanner 59a9d7684ef74d7782385fb753644bbe for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is aff82a89f7894c9d8597ace5f998b5be
0114 03:03:45.120147 (+ 442us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":24448}
W20250114 03:03:45.120678 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.156s user 0.000s sys 0.000s
W20250114 03:03:45.120860 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.156s user 0.000s sys 0.000s
I20250114 03:03:45.535154 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:45.931460 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:45.931609 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:45.932148 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:45.932562 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:45.932883 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:45.933239 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:45.933562 3584 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=24, attempt_no=0}) took 1927 ms. Trace:
I20250114 03:03:45.933632 3584 rpcz_store.cc:276] 0114 03:03:44.005822 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:44.005906 (+ 84us) service_pool.cc:225] Handling call
0114 03:03:45.933547 (+1927641us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:44.632782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:44.632939 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:44.632945 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:44.632947 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:44.633039 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:44.633049 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:44.633052 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:44.633102 (+ 50us) tablet.cc:698] Row locks acquired
0114 03:03:44.633104 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:44.633145 (+ 41us) write_op.cc:270] Start()
0114 03:03:45.268281 (+635136us) write_op.cc:276] Timestamp: P: 1736823824633094 usec, L: 0
0114 03:03:45.268285 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:45.268462 (+ 177us) log.cc:844] Serialized 354 byte log entry
0114 03:03:45.932552 (+664090us) op_driver.cc:464] REPLICATION: finished
0114 03:03:45.933204 (+ 652us) write_op.cc:301] APPLY: starting
0114 03:03:45.933221 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:45.933254 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:45.933266 (+ 12us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:45.933321 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:45.933348 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:45.933352 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:45.933412 (+ 60us) log.cc:844] Serialized 22 byte log entry
0114 03:03:45.933464 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:45.933476 (+ 12us) write_op.cc:454] Released schema lock
0114 03:03:45.933481 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":602,"num_ops":1,"prepare.queue_time_us":626562,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":635741,"replication_time_us":664235,"thread_start_us":105,"threads_started":1}]]}
I20250114 03:03:46.102885 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:46.103042 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:46.103722 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:46.104169 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:46.104689 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:46.105170 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:46.267048 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:46.268466 3584 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=25, attempt_no=0}) took 1147 ms. Trace:
I20250114 03:03:46.268548 3584 rpcz_store.cc:276] 0114 03:03:45.120801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:45.120860 (+ 59us) service_pool.cc:225] Handling call
0114 03:03:46.268452 (+1147592us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:45.268566 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:45.268721 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:45.268725 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:45.268727 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:45.268814 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:45.268850 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:45.268855 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:45.268902 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:03:45.268904 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:45.268948 (+ 44us) write_op.cc:270] Start()
0114 03:03:45.603107 (+334159us) write_op.cc:276] Timestamp: P: 1736823825268906 usec, L: 0
0114 03:03:45.603111 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:45.603413 (+ 302us) log.cc:844] Serialized 354 byte log entry
0114 03:03:46.267949 (+664536us) op_driver.cc:464] REPLICATION: finished
0114 03:03:46.268049 (+ 100us) write_op.cc:301] APPLY: starting
0114 03:03:46.268070 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:46.268102 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:46.268122 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:46.268186 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:46.268220 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:46.268226 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:03:46.268290 (+ 64us) log.cc:844] Serialized 22 byte log entry
0114 03:03:46.268362 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:46.268376 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:46.268381 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":48,"num_ops":1,"prepare.queue_time_us":147387,"prepare.run_cpu_time_us":847,"prepare.run_wall_time_us":334909,"replication_time_us":664806}]]}
W20250114 03:03:46.269729 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.998s user 0.000s sys 0.000s
W20250114 03:03:46.269877 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.999s user 0.001s sys 0.000s
I20250114 03:03:46.476517 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:46.476662 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:46.477355 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:46.477766 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:46.477946 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:46.478497 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:46.478792 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:46.744009 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:46.744154 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:46.744794 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:46.745025 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:46.745959 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 29 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:46.746783 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:46.747056 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:47.192319 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:47.192466 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:47.193045 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:47.193476 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:47.193539 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:47.194074 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:47.194233 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:47.499392 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:47.499544 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:47.500061 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:47.500450 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:47.500573 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:47.501044 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:47.501345 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:47.529481 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:47.529651 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:47.530126 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:47.530566 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:47.530606 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 32 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:47.531018 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:47.531675 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:47.605201 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:47.605747 3361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:55207 (request call id 3671) took 1501 ms. Trace:
I20250114 03:03:47.605837 3361 rpcz_store.cc:276] 0114 03:03:46.104194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:46.104250 (+ 56us) service_pool.cc:225] Handling call
0114 03:03:46.104411 (+ 161us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:03:46.104512 (+ 101us) raft_consensus.cc:1529] Early marking committed up to index 26
0114 03:03:46.104679 (+ 167us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:03:46.104870 (+ 191us) write_op.cc:270] Start()
0114 03:03:46.609023 (+504153us) write_op.cc:276] Timestamp: P: 1736823825268906 usec, L: 0
0114 03:03:46.609408 (+ 385us) write_op.cc:270] Start()
0114 03:03:47.604561 (+995153us) write_op.cc:276] Timestamp: P: 1736823825935254 usec, L: 0
0114 03:03:47.605011 (+ 450us) log.cc:844] Serialized 708 byte log entry
0114 03:03:47.605110 (+ 99us) raft_consensus.cc:1640] Marking committed up to 26
0114 03:03:47.605118 (+ 8us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:03:47.605147 (+ 29us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:03:47.605688 (+ 541us) raft_consensus.cc:1680] finished
0114 03:03:47.605690 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:03:47.605732 (+ 42us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:46.609342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:46.609450 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:46.609454 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:46.609455 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:46.609520 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:46.609529 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:46.609532 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:46.609567 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:03:46.609568 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:03:47.604941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:47.605065 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:47.605071 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:47.605073 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:47.605170 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:47.605203 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:47.605206 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:47.605254 (+ 48us) tablet.cc:698] Row locks acquired
0114 03:03:47.605255 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":49,"threads_started":1,"wal-append.queue_time_us":192,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":189,"prepare.run_cpu_time_us":310,"prepare.run_wall_time_us":311,"thread_start_us":83,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":27,"prepare.queue_time_us":225,"prepare.run_cpu_time_us":405,"prepare.run_wall_time_us":406,"thread_start_us":79,"threads_started":1}]]}
I20250114 03:03:48.138604 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:48.138742 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:48.139267 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:48.139667 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:48.139698 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:48.140306 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:48.140542 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:48.462917 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:48.463085 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:48.463600 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:48.464119 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:48.464463 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:48.465823 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:48.466022 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:03:48.466037 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.783s user 0.001s sys 0.000s
W20250114 03:03:48.466140 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.783s user 0.001s sys 0.000s
I20250114 03:03:49.408382 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:49.408437 3361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:55207 (request call id 3672) took 1801 ms. Trace:
I20250114 03:03:49.408516 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:49.408511 3361 rpcz_store.cc:276] 0114 03:03:47.607422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:47.607470 (+ 48us) service_pool.cc:225] Handling call
0114 03:03:47.607666 (+ 196us) raft_consensus.cc:1466] Updating replica for 5 ops
0114 03:03:47.607759 (+ 93us) raft_consensus.cc:1529] Early marking committed up to index 28
0114 03:03:47.607974 (+ 215us) raft_consensus.cc:1534] Triggering prepare for 5 ops
0114 03:03:47.608106 (+ 132us) write_op.cc:270] Start()
0114 03:03:48.307251 (+699145us) write_op.cc:276] Timestamp: P: 1736823826270718 usec, L: 0
0114 03:03:48.307647 (+ 396us) write_op.cc:270] Start()
0114 03:03:48.679781 (+372134us) write_op.cc:276] Timestamp: P: 1736823826479907 usec, L: 0
0114 03:03:48.680043 (+ 262us) write_op.cc:270] Start()
0114 03:03:49.053306 (+373263us) write_op.cc:276] Timestamp: P: 1736823827107713 usec, L: 0
0114 03:03:49.053639 (+ 333us) write_op.cc:270] Start()
0114 03:03:49.119798 (+ 66159us) write_op.cc:276] Timestamp: P: 1736823827312384 usec, L: 0
0114 03:03:49.120089 (+ 291us) write_op.cc:270] Start()
0114 03:03:49.407231 (+287142us) write_op.cc:276] Timestamp: P: 1736823827502560 usec, L: 0
0114 03:03:49.407649 (+ 418us) log.cc:844] Serialized 1770 byte log entry
0114 03:03:49.407803 (+ 154us) raft_consensus.cc:1640] Marking committed up to 32
0114 03:03:49.408334 (+ 531us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:03:49.408362 (+ 28us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:03:49.408368 (+ 6us) raft_consensus.cc:1680] finished
0114 03:03:49.408369 (+ 1us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:03:49.408418 (+ 49us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:48.307629 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:48.307756 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:48.307763 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:03:48.307767 (+ 4us) tablet.cc:659] Decoding operations
0114 03:03:48.307836 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:48.307846 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:48.307848 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:48.307885 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:03:48.307886 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:49.407839 (+1099953us) op_driver.cc:464] REPLICATION: finished
0114 03:03:49.408055 (+ 216us) write_op.cc:301] APPLY: starting
0114 03:03:49.408072 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:49.408115 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:49.408117 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:49.408179 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:49.408209 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:49.408232 (+ 23us) write_op.cc:312] APPLY: finished
0114 03:03:49.408298 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:03:49.408437 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:49.408451 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:49.408458 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0114 03:03:49.408547 (+ 89us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:49.408549 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:03:48.679958 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:48.680075 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:48.680079 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:48.680081 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:48.680153 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:48.680162 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:48.680164 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:48.680201 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:03:48.680202 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:49.407968 (+727766us) op_driver.cc:464] REPLICATION: finished
0114 03:03:49.408196 (+ 228us) write_op.cc:301] APPLY: starting
0114 03:03:49.408212 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:49.408233 (+ 21us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:49.408235 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:49.408308 (+ 73us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:49.408332 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:49.408353 (+ 21us) write_op.cc:312] APPLY: finished
0114 03:03:49.408619 (+ 266us) log.cc:844] Serialized 22 byte log entry
0114 03:03:49.408667 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:49.408676 (+ 9us) write_op.cc:454] Released schema lock
0114 03:03:49.408680 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 03:03:49.053561 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:49.053695 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:49.053699 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:49.053700 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:49.053768 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:49.053777 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:49.053779 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:49.053814 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:03:49.053815 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:49.408085 (+354270us) op_driver.cc:464] REPLICATION: finished
0114 03:03:49.408480 (+ 395us) write_op.cc:301] APPLY: starting
0114 03:03:49.408492 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:49.408508 (+ 16us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:49.408509 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:49.408542 (+ 33us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:49.408564 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:49.408580 (+ 16us) write_op.cc:312] APPLY: finished
0114 03:03:49.408750 (+ 170us) log.cc:844] Serialized 22 byte log entry
Related trace 'op':
0114 03:03:49.119990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:49.120104 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:49.120111 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:03:49.120121 (+ 10us) tablet.cc:659] Decoding operations
0114 03:03:49.120190 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:49.120200 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:49.120202 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:49.120240 (+ 38us) tablet.cc:698] Row locks acquired
0114 03:03:49.120241 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:49.408221 (+287980us) op_driver.cc:464] REPLICATION: finished
0114 03:03:49.408596 (+ 375us) write_op.cc:301] APPLY: starting
0114 03:03:49.408606 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:49.408622 (+ 16us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:49.408623 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:49.408692 (+ 69us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:49.408711 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:49.408721 (+ 10us) write_op.cc:312] APPLY: finished
0114 03:03:49.408759 (+ 38us) log.cc:844] Serialized 22 byte log entry
0114 03:03:49.408786 (+ 27us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:49.408794 (+ 8us) write_op.cc:454] Released schema lock
0114 03:03:49.408797 (+ 3us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0114 03:03:49.407416 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:49.407548 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:49.407553 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:49.407555 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:49.407631 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:49.407641 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:49.407644 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:49.407690 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:03:49.407691 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":90,"threads_started":1,"wal-append.queue_time_us":182,"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":485,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":217,"prepare.run_cpu_time_us":363,"prepare.run_wall_time_us":362,"replication_time_us":1799716,"thread_start_us":181,"threads_started":2}],["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":338,"apply.run_wall_time_us":557,"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":312,"prepare.run_wall_time_us":313,"replication_time_us":1100309,"thread_start_us":49,"threads_started":1}],["op",{"apply.queue_time_us":359,"apply.run_cpu_time_us":275,"apply.run_wall_time_us":546,"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":311,"prepare.run_wall_time_us":312,"replication_time_us":728033,"thread_start_us":78,"threads_started":1}],["op",{"apply.queue_time_us":353,"apply.run_cpu_time_us":268,"apply.run_wall_time_us":268,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":306,"prepare.run_wall_time_us":307,"replication_time_us":354572,"thread_start_us":67,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":344,"prepare.run_wall_time_us":344}]]}
I20250114 03:03:49.409444 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:49.410001 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:49.410001 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:49.410395 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:49.812937 3559 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:49.813088 3559 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:49.813596 3559 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:49.814038 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:49.814150 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:49.814632 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:49.815574 3607 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=33, attempt_no=0}) took 1673 ms. Trace:
I20250114 03:03:49.815688 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 77) took 1198 ms. Trace:
I20250114 03:03:49.815815 3056 rpcz_store.cc:276] 0114 03:03:48.617073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:48.617125 (+ 52us) service_pool.cc:225] Handling call
0114 03:03:48.617292 (+ 167us) tablet_service.cc:2780] Created scanner 5c54a385e7bd46ca88313e7c0d945c87 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 65f418e1043242cfa8ad0d5fa45dc55b
0114 03:03:48.617482 (+ 190us) tablet_service.cc:2920] Creating iterator
0114 03:03:48.617517 (+ 35us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:48.857317 (+239800us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:49.814783 (+957466us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1197255 microseconds
0114 03:03:49.814876 (+ 93us) tablet_service.cc:2945] Iterator created
0114 03:03:49.815185 (+ 309us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:49.815209 (+ 24us) tablet_service.cc:3010] has_more: true
0114 03:03:49.815239 (+ 30us) tablet_service.cc:3027] Continuing scan request
0114 03:03:49.815282 (+ 43us) tablet_service.cc:3091] Found scanner 5c54a385e7bd46ca88313e7c0d945c87 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 65f418e1043242cfa8ad0d5fa45dc55b
0114 03:03:49.815675 (+ 393us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":28160}
I20250114 03:03:49.815683 3607 rpcz_store.cc:276] 0114 03:03:48.141982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:48.142027 (+ 45us) service_pool.cc:225] Handling call
0114 03:03:49.815559 (+1673532us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:48.414766 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:48.414882 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:48.414902 (+ 20us) write_op.cc:435] Acquired schema lock
0114 03:03:48.414904 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:48.414985 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:48.414995 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:48.414998 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:48.415037 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:03:48.415038 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:48.415071 (+ 33us) write_op.cc:270] Start()
0114 03:03:48.857192 (+442121us) write_op.cc:276] Timestamp: P: 1736823828415010 usec, L: 0
0114 03:03:48.857195 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:03:48.857430 (+ 235us) log.cc:844] Serialized 354 byte log entry
0114 03:03:49.814174 (+956744us) op_driver.cc:464] REPLICATION: finished
0114 03:03:49.814439 (+ 265us) write_op.cc:301] APPLY: starting
0114 03:03:49.814457 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:49.814500 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:49.814513 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:49.814570 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:49.814597 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:49.814601 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:49.814673 (+ 72us) log.cc:844] Serialized 22 byte log entry
0114 03:03:49.815467 (+ 794us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:49.815481 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:49.815487 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":212,"num_ops":1,"prepare.queue_time_us":272415,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":442716,"replication_time_us":956948,"thread_start_us":66,"threads_started":1}]]}
I20250114 03:03:49.816406 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 78) took 1199 ms. Trace:
W20250114 03:03:49.816517 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.200s user 0.001s sys 0.000s
I20250114 03:03:49.816473 3055 rpcz_store.cc:276] 0114 03:03:48.617145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:48.617192 (+ 47us) service_pool.cc:225] Handling call
0114 03:03:48.617307 (+ 115us) tablet_service.cc:2780] Created scanner 0ec7a679b3e04d9f981b9eb14380b8ba for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is cbaccba3ad584df8875dd5281f3c3137
0114 03:03:48.617479 (+ 172us) tablet_service.cc:2920] Creating iterator
0114 03:03:48.617522 (+ 43us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:48.857321 (+239799us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:49.815504 (+958183us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1197972 microseconds
0114 03:03:49.815617 (+ 113us) tablet_service.cc:2945] Iterator created
0114 03:03:49.815938 (+ 321us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:49.815961 (+ 23us) tablet_service.cc:3010] has_more: true
0114 03:03:49.815994 (+ 33us) tablet_service.cc:3027] Continuing scan request
0114 03:03:49.816031 (+ 37us) tablet_service.cc:3091] Found scanner 0ec7a679b3e04d9f981b9eb14380b8ba for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is cbaccba3ad584df8875dd5281f3c3137
0114 03:03:49.816396 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":25728}
W20250114 03:03:49.816885 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.200s user 0.001s sys 0.000s
I20250114 03:03:49.851616 3559 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:49.853101 3607 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=34, attempt_no=0}) took 1386 ms. Trace:
I20250114 03:03:49.853199 3607 rpcz_store.cc:276] 0114 03:03:48.466454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:48.466517 (+ 63us) service_pool.cc:225] Handling call
0114 03:03:49.853084 (+1386567us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:48.857506 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:48.857621 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:48.857629 (+ 8us) write_op.cc:435] Acquired schema lock
0114 03:03:48.857632 (+ 3us) tablet.cc:659] Decoding operations
0114 03:03:48.857708 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:48.857716 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:48.857718 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:48.857756 (+ 38us) tablet.cc:698] Row locks acquired
0114 03:03:48.857757 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:48.857795 (+ 38us) write_op.cc:270] Start()
0114 03:03:49.238973 (+381178us) write_op.cc:276] Timestamp: P: 1736823828857742 usec, L: 0
0114 03:03:49.238977 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:49.239184 (+ 207us) log.cc:844] Serialized 354 byte log entry
0114 03:03:49.852557 (+613373us) op_driver.cc:464] REPLICATION: finished
0114 03:03:49.852670 (+ 113us) write_op.cc:301] APPLY: starting
0114 03:03:49.852689 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:49.852719 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:49.852735 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:49.852790 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:49.852814 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:49.852845 (+ 31us) write_op.cc:312] APPLY: finished
0114 03:03:49.852913 (+ 68us) log.cc:844] Serialized 22 byte log entry
0114 03:03:49.852991 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:49.853004 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:49.853009 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":58,"num_ops":1,"prepare.queue_time_us":390642,"prepare.run_cpu_time_us":658,"prepare.run_wall_time_us":381736,"replication_time_us":613548}]]}
I20250114 03:03:50.484690 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:50.485167 3361 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:55207 (request call id 3673) took 1074 ms. Trace:
I20250114 03:03:50.485244 3361 rpcz_store.cc:276] 0114 03:03:49.410294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:49.410335 (+ 41us) service_pool.cc:225] Handling call
0114 03:03:49.410508 (+ 173us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 03:03:49.410588 (+ 80us) raft_consensus.cc:1529] Early marking committed up to index 33
0114 03:03:49.410661 (+ 73us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 03:03:49.410725 (+ 64us) write_op.cc:270] Start()
0114 03:03:49.458897 (+ 48172us) write_op.cc:276] Timestamp: P: 1736823827584281 usec, L: 0
0114 03:03:49.459215 (+ 318us) write_op.cc:270] Start()
0114 03:03:50.438410 (+979195us) write_op.cc:276] Timestamp: P: 1736823828415010 usec, L: 0
0114 03:03:50.438770 (+ 360us) write_op.cc:270] Start()
0114 03:03:50.483952 (+ 45182us) write_op.cc:276] Timestamp: P: 1736823828857742 usec, L: 0
0114 03:03:50.484316 (+ 364us) log.cc:844] Serialized 1062 byte log entry
0114 03:03:50.484465 (+ 149us) raft_consensus.cc:1640] Marking committed up to 34
0114 03:03:50.484623 (+ 158us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:03:50.484647 (+ 24us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:03:50.485099 (+ 452us) raft_consensus.cc:1680] finished
0114 03:03:50.485100 (+ 1us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:03:50.485150 (+ 50us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:49.459090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:49.459227 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:49.459233 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:49.459236 (+ 3us) tablet.cc:659] Decoding operations
0114 03:03:49.459309 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:49.459319 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:49.459321 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:49.459360 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:03:49.459361 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:50.484502 (+1025141us) op_driver.cc:464] REPLICATION: finished
0114 03:03:50.484718 (+ 216us) write_op.cc:301] APPLY: starting
0114 03:03:50.484736 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:50.484771 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:50.484772 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:50.484853 (+ 81us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:50.484887 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:50.484914 (+ 27us) write_op.cc:312] APPLY: finished
0114 03:03:50.484977 (+ 63us) log.cc:844] Serialized 22 byte log entry
0114 03:03:50.485016 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:50.485027 (+ 11us) write_op.cc:454] Released schema lock
0114 03:03:50.485031 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0114 03:03:50.485114 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:50.485116 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:03:50.438749 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:50.438871 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:50.438874 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:50.438876 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:50.438968 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:50.438978 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:50.438981 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:50.439018 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:03:50.439019 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:03:50.484137 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:50.484289 (+ 152us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:50.484295 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:50.484296 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:50.484369 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:50.484383 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:50.484389 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:50.484438 (+ 49us) tablet.cc:698] Row locks acquired
0114 03:03:50.484440 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":87,"threads_started":1,"wal-append.queue_time_us":258,"child_traces":[["op",{"apply.queue_time_us":153,"apply.run_cpu_time_us":393,"apply.run_wall_time_us":393,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":352,"prepare.run_wall_time_us":353,"replication_time_us":1073759,"thread_start_us":72,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":224,"prepare.run_cpu_time_us":336,"prepare.run_wall_time_us":336,"thread_start_us":83,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":59,"prepare.run_cpu_time_us":361,"prepare.run_wall_time_us":361}]]}
I20250114 03:03:51.010915 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:51.011065 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:51.011605 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:51.012023 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:51.012086 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:51.012648 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:51.013557 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 81) took 1045 ms. Trace:
I20250114 03:03:51.013602 3618 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=35, attempt_no=0}) took 1196 ms. Trace:
I20250114 03:03:51.013622 3055 rpcz_store.cc:276] 0114 03:03:49.967630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:49.967696 (+ 66us) service_pool.cc:225] Handling call
0114 03:03:49.967865 (+ 169us) tablet_service.cc:2780] Created scanner c86b7d45eaa24e6ca06a9838bfaf3067 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 56afd08529bd4f4d9eebbca53a2dcdec
0114 03:03:49.968095 (+ 230us) tablet_service.cc:2920] Creating iterator
0114 03:03:49.968133 (+ 38us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:50.564171 (+596038us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:51.012674 (+448503us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1044531 microseconds
0114 03:03:51.012762 (+ 88us) tablet_service.cc:2945] Iterator created
0114 03:03:51.013150 (+ 388us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:51.013174 (+ 24us) tablet_service.cc:3010] has_more: true
0114 03:03:51.013204 (+ 30us) tablet_service.cc:3027] Continuing scan request
0114 03:03:51.013245 (+ 41us) tablet_service.cc:3091] Found scanner c86b7d45eaa24e6ca06a9838bfaf3067 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 56afd08529bd4f4d9eebbca53a2dcdec
0114 03:03:51.013544 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":35840}
I20250114 03:03:51.013716 3618 rpcz_store.cc:276] 0114 03:03:49.817196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:49.817253 (+ 57us) service_pool.cc:225] Handling call
0114 03:03:51.013587 (+1196334us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:49.817673 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:49.817761 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:49.817764 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:49.817765 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:49.817816 (+ 51us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:49.817822 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:49.817824 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:49.817853 (+ 29us) tablet.cc:698] Row locks acquired
0114 03:03:49.817854 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:49.817880 (+ 26us) write_op.cc:270] Start()
0114 03:03:50.564028 (+746148us) write_op.cc:276] Timestamp: P: 1736823829817840 usec, L: 0
0114 03:03:50.564033 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:03:50.564275 (+ 242us) log.cc:844] Serialized 354 byte log entry
0114 03:03:51.011979 (+447704us) op_driver.cc:464] REPLICATION: finished
0114 03:03:51.012213 (+ 234us) write_op.cc:301] APPLY: starting
0114 03:03:51.012230 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:51.012260 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:51.012272 (+ 12us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:51.012333 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:51.012494 (+ 161us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:51.012503 (+ 9us) write_op.cc:312] APPLY: finished
0114 03:03:51.012573 (+ 70us) log.cc:844] Serialized 22 byte log entry
0114 03:03:51.013495 (+ 922us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:51.013506 (+ 11us) write_op.cc:454] Released schema lock
0114 03:03:51.013513 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":187,"mutex_wait_us":3,"num_ops":1,"prepare.queue_time_us":147,"prepare.run_cpu_time_us":724,"prepare.run_wall_time_us":746793,"raft.queue_time_us":82,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":447917,"thread_start_us":256,"threads_started":3}]]}
I20250114 03:03:51.014394 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 82) took 1046 ms. Trace:
W20250114 03:03:51.014520 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.047s user 0.001s sys 0.000s
I20250114 03:03:51.014492 3056 rpcz_store.cc:276] 0114 03:03:49.967739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:49.967780 (+ 41us) service_pool.cc:225] Handling call
0114 03:03:49.967874 (+ 94us) tablet_service.cc:2780] Created scanner 6140cb3fb944479ba951f001a94afa43 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 69156b4d486d4e21ac8094d43985a160
0114 03:03:49.968095 (+ 221us) tablet_service.cc:2920] Creating iterator
0114 03:03:49.968131 (+ 36us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:50.564171 (+596040us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:51.013556 (+449385us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1045416 microseconds
0114 03:03:51.013653 (+ 97us) tablet_service.cc:2945] Iterator created
0114 03:03:51.013940 (+ 287us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:51.013964 (+ 24us) tablet_service.cc:3010] has_more: true
0114 03:03:51.014000 (+ 36us) tablet_service.cc:3027] Continuing scan request
0114 03:03:51.014049 (+ 49us) tablet_service.cc:3091] Found scanner 6140cb3fb944479ba951f001a94afa43 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 69156b4d486d4e21ac8094d43985a160
0114 03:03:51.014385 (+ 336us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":17664}
W20250114 03:03:51.014977 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.048s user 0.000s sys 0.000s
I20250114 03:03:51.198746 3616 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:51.198902 3616 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:51.199432 3616 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:51.199873 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:51.199888 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:51.200464 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:51.766829 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:51.768533 3626 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=36, attempt_no=0}) took 1913 ms. Trace:
I20250114 03:03:51.768617 3626 rpcz_store.cc:276] 0114 03:03:49.854689 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:49.854769 (+ 80us) service_pool.cc:225] Handling call
0114 03:03:51.768519 (+1913750us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:50.564492 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:50.564627 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:50.564654 (+ 27us) write_op.cc:435] Acquired schema lock
0114 03:03:50.564655 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:50.564743 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:50.564752 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:50.564754 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:50.564790 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:03:50.564791 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:50.564840 (+ 49us) write_op.cc:270] Start()
0114 03:03:50.584094 (+ 19254us) write_op.cc:276] Timestamp: P: 1736823830564763 usec, L: 0
0114 03:03:50.584098 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:50.584318 (+ 220us) log.cc:844] Serialized 354 byte log entry
0114 03:03:51.767881 (+1183563us) op_driver.cc:464] REPLICATION: finished
0114 03:03:51.768098 (+ 217us) write_op.cc:301] APPLY: starting
0114 03:03:51.768116 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:51.768147 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:51.768162 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:51.768228 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:51.768259 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:51.768265 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:03:51.768344 (+ 79us) log.cc:844] Serialized 22 byte log entry
0114 03:03:51.768421 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:51.768438 (+ 17us) write_op.cc:454] Released schema lock
0114 03:03:51.768442 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":155,"num_ops":1,"prepare.queue_time_us":709332,"prepare.run_cpu_time_us":746,"prepare.run_wall_time_us":19898,"replication_time_us":1183743,"thread_start_us":76,"threads_started":1}]]}
I20250114 03:03:52.068428 3616 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:52.490464 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:52.490600 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:52.491086 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:52.491588 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:52.491588 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:52.492075 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:52.492269 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:52.493099 3633 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=37, attempt_no=0}) took 1478 ms. Trace:
I20250114 03:03:52.493217 3633 rpcz_store.cc:276] 0114 03:03:51.015031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:51.015092 (+ 61us) service_pool.cc:225] Handling call
0114 03:03:52.493084 (+1477992us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:51.015413 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:51.015483 (+ 70us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:51.015486 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:51.015487 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:51.015540 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:51.015547 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:51.015549 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:51.015577 (+ 28us) tablet.cc:698] Row locks acquired
0114 03:03:51.015578 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:51.015603 (+ 25us) write_op.cc:270] Start()
0114 03:03:51.244789 (+229186us) write_op.cc:276] Timestamp: P: 1736823831015536 usec, L: 0
0114 03:03:51.244796 (+ 7us) op_driver.cc:348] REPLICATION: starting
0114 03:03:51.245116 (+ 320us) log.cc:844] Serialized 354 byte log entry
0114 03:03:52.492470 (+1247354us) op_driver.cc:464] REPLICATION: finished
0114 03:03:52.492665 (+ 195us) write_op.cc:301] APPLY: starting
0114 03:03:52.492683 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:52.492718 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:52.492736 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:52.492797 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:52.492843 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:52.492849 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:03:52.492915 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:03:52.492993 (+ 78us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:52.493007 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:52.493012 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":135,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":229758,"replication_time_us":1247643,"thread_start_us":61,"threads_started":1}]]}
I20250114 03:03:52.494028 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 84) took 1328 ms. Trace:
I20250114 03:03:52.494100 3056 rpcz_store.cc:276] 0114 03:03:51.165790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:51.165867 (+ 77us) service_pool.cc:225] Handling call
0114 03:03:51.166045 (+ 178us) tablet_service.cc:2780] Created scanner db367055c91f4bedb59c79edb4af275a for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is f2a8de857bbd40669b3c91bea797bee1
0114 03:03:51.166245 (+ 200us) tablet_service.cc:2920] Creating iterator
0114 03:03:51.166283 (+ 38us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:51.245018 (+ 78735us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:52.493199 (+1248181us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1326909 microseconds
0114 03:03:52.493246 (+ 47us) tablet_service.cc:2945] Iterator created
0114 03:03:52.493484 (+ 238us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:52.493510 (+ 26us) tablet_service.cc:3010] has_more: true
0114 03:03:52.493545 (+ 35us) tablet_service.cc:3027] Continuing scan request
0114 03:03:52.493594 (+ 49us) tablet_service.cc:3091] Found scanner db367055c91f4bedb59c79edb4af275a for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is f2a8de857bbd40669b3c91bea797bee1
0114 03:03:52.494017 (+ 423us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":29440}
I20250114 03:03:52.494450 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 85) took 1328 ms. Trace:
I20250114 03:03:52.494520 3055 rpcz_store.cc:276] 0114 03:03:51.165866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:51.165926 (+ 60us) service_pool.cc:225] Handling call
0114 03:03:51.166057 (+ 131us) tablet_service.cc:2780] Created scanner 87f52bcb48c0445d80c550b8de2e7f5e for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 63b183caafcc4592a23bb460f3be73c7
0114 03:03:51.166238 (+ 181us) tablet_service.cc:2920] Creating iterator
0114 03:03:51.166281 (+ 43us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:51.244999 (+ 78718us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:52.493039 (+1248040us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1326744 microseconds
0114 03:03:52.493137 (+ 98us) tablet_service.cc:2945] Iterator created
0114 03:03:52.493484 (+ 347us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:52.493508 (+ 24us) tablet_service.cc:3010] has_more: true
0114 03:03:52.493545 (+ 37us) tablet_service.cc:3027] Continuing scan request
0114 03:03:52.493594 (+ 49us) tablet_service.cc:3091] Found scanner 87f52bcb48c0445d80c550b8de2e7f5e for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 63b183caafcc4592a23bb460f3be73c7
0114 03:03:52.494443 (+ 849us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":23680}
W20250114 03:03:52.494895 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.330s user 0.001s sys 0.000s
W20250114 03:03:52.494961 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.330s user 0.000s sys 0.000s
I20250114 03:03:53.069545 3616 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:53.069710 3616 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:53.070284 3616 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:53.070786 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:53.071550 3639 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=38, attempt_no=0}) took 1301 ms. Trace:
I20250114 03:03:53.071935 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:53.071676 3639 rpcz_store.cc:276] 0114 03:03:51.769926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:51.769989 (+ 63us) service_pool.cc:225] Handling call
0114 03:03:53.071533 (+1301544us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:51.770484 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:51.770567 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:51.770569 (+ 2us) write_op.cc:435] Acquired schema lock
0114 03:03:51.770570 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:51.770623 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:51.770633 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:51.770637 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:51.770673 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:03:51.770675 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:51.770707 (+ 32us) write_op.cc:270] Start()
0114 03:03:51.933875 (+163168us) write_op.cc:276] Timestamp: P: 1736823831770673 usec, L: 0
0114 03:03:51.933881 (+ 6us) op_driver.cc:348] REPLICATION: starting
0114 03:03:51.934098 (+ 217us) log.cc:844] Serialized 354 byte log entry
0114 03:03:53.070813 (+1136715us) op_driver.cc:464] REPLICATION: finished
0114 03:03:53.071137 (+ 324us) write_op.cc:301] APPLY: starting
0114 03:03:53.071156 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:53.071188 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:53.071204 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:53.071273 (+ 69us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:53.071308 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:53.071312 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:53.071387 (+ 75us) log.cc:844] Serialized 22 byte log entry
0114 03:03:53.071444 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:53.071457 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:53.071463 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":261,"num_ops":1,"prepare.queue_time_us":173,"prepare.run_cpu_time_us":605,"prepare.run_wall_time_us":163683,"replication_time_us":1136890,"thread_start_us":158,"threads_started":2}]]}
I20250114 03:03:53.072268 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:53.072417 3616 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:53.370169 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:53.370326 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:53.370872 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:53.371366 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:53.371420 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:53.371824 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:53.567677 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:53.569324 3639 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=39, attempt_no=0}) took 1074 ms. Trace:
I20250114 03:03:53.569419 3639 rpcz_store.cc:276] 0114 03:03:52.494884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:52.494945 (+ 61us) service_pool.cc:225] Handling call
0114 03:03:53.569309 (+1074364us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:52.495402 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:52.495494 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:52.495497 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:03:52.495498 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:52.495550 (+ 52us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:52.495555 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:52.495557 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:52.495586 (+ 29us) tablet.cc:698] Row locks acquired
0114 03:03:52.495586 (+ 0us) write_op.cc:260] PREPARE: finished
0114 03:03:52.495612 (+ 26us) write_op.cc:270] Start()
0114 03:03:53.091747 (+596135us) write_op.cc:276] Timestamp: P: 1736823832495556 usec, L: 0
0114 03:03:53.091751 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:53.091974 (+ 223us) log.cc:844] Serialized 354 byte log entry
0114 03:03:53.568761 (+476787us) op_driver.cc:464] REPLICATION: finished
0114 03:03:53.568882 (+ 121us) write_op.cc:301] APPLY: starting
0114 03:03:53.568901 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:53.568929 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:53.568945 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:53.569008 (+ 63us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:53.569041 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:53.569045 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:53.569111 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:03:53.569211 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:53.569229 (+ 18us) write_op.cc:454] Released schema lock
0114 03:03:53.569234 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":54,"num_ops":1,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":587,"prepare.run_wall_time_us":596656,"raft.queue_time_us":36,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":162,"replication_time_us":476974,"thread_start_us":56,"threads_started":1}]]}
W20250114 03:03:53.570683 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.925s user 0.001s sys 0.000s
W20250114 03:03:53.570698 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.925s user 0.001s sys 0.000s
I20250114 03:03:53.588673 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:53.588811 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:53.589268 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:53.589747 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:53.589757 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:53.590195 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:53.590411 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:53.695505 3616 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:53.695647 3616 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:53.696257 3616 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:53.696703 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:53.696791 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:53.697232 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:53.697460 3616 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:53.962050 3616 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:53.962204 3616 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:53.962561 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:53.962713 3616 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:53.962680 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:53.963090 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:53.963126 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:53.963402 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:53.963519 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:53.963812 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:53.963115 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 41 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:53.964066 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:53.964488 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:53.964674 3616 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:54.293298 3616 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:54.293431 3616 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:54.293996 3616 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:54.294468 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:54.294622 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:54.295218 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:54.295397 3616 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:54.295732 3647 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=40, attempt_no=0}) took 1222 ms. Trace:
I20250114 03:03:54.295859 3647 rpcz_store.cc:276] 0114 03:03:53.073397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:53.073453 (+ 56us) service_pool.cc:225] Handling call
0114 03:03:54.295715 (+1222262us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:53.092089 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:53.092209 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:53.092215 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:53.092216 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:53.092291 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:53.092300 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:53.092303 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:53.092344 (+ 41us) tablet.cc:698] Row locks acquired
0114 03:03:53.092345 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:53.092376 (+ 31us) write_op.cc:270] Start()
0114 03:03:54.084518 (+992142us) write_op.cc:276] Timestamp: P: 1736823833092318 usec, L: 0
0114 03:03:54.084521 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:03:54.084776 (+ 255us) log.cc:844] Serialized 354 byte log entry
0114 03:03:54.294626 (+209850us) op_driver.cc:464] REPLICATION: finished
0114 03:03:54.295020 (+ 394us) write_op.cc:301] APPLY: starting
0114 03:03:54.295040 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:54.295076 (+ 36us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:54.295090 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:54.295152 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:54.295182 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:54.295187 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:54.295257 (+ 70us) log.cc:844] Serialized 22 byte log entry
0114 03:03:54.295562 (+ 305us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:54.295576 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:54.295582 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":332,"num_ops":1,"prepare.queue_time_us":18260,"prepare.run_cpu_time_us":704,"prepare.run_wall_time_us":992774,"replication_time_us":210077,"thread_start_us":92,"threads_started":1}]]}
W20250114 03:03:54.297227 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.576s user 0.001s sys 0.000s
W20250114 03:03:54.297227 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.576s user 0.001s sys 0.000s
I20250114 03:03:54.394344 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:54.394505 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:54.395211 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:54.395452 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:54.395612 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:54.396049 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:54.396267 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:55.354827 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:55.354974 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.355453 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:55.355937 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:55.355937 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:55.356415 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:55.356855 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:55.357088 3655 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=41, attempt_no=0}) took 1786 ms. Trace:
I20250114 03:03:55.357192 3655 rpcz_store.cc:276] 0114 03:03:53.570839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:53.570967 (+ 128us) service_pool.cc:225] Handling call
0114 03:03:55.357074 (+1786107us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:54.084907 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:54.085022 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:54.085027 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:54.085028 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:54.085110 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:54.085121 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:54.085127 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:54.085174 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:03:54.085175 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:54.085213 (+ 38us) write_op.cc:270] Start()
0114 03:03:54.852359 (+767146us) write_op.cc:276] Timestamp: P: 1736823834085140 usec, L: 0
0114 03:03:54.852362 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:03:54.852576 (+ 214us) log.cc:844] Serialized 354 byte log entry
0114 03:03:55.356415 (+503839us) op_driver.cc:464] REPLICATION: finished
0114 03:03:55.356647 (+ 232us) write_op.cc:301] APPLY: starting
0114 03:03:55.356664 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:55.356693 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:55.356707 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:55.356768 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:55.356791 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:55.356794 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:03:55.356886 (+ 92us) log.cc:844] Serialized 22 byte log entry
0114 03:03:55.356977 (+ 91us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:55.356991 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:55.356996 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":186,"num_ops":1,"prepare.queue_time_us":513620,"prepare.run_cpu_time_us":655,"prepare.run_wall_time_us":767725,"replication_time_us":504023,"thread_start_us":82,"threads_started":1}]]}
W20250114 03:03:55.358637 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.911s user 0.001s sys 0.000s
W20250114 03:03:55.358646 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.911s user 0.000s sys 0.000s
I20250114 03:03:55.368531 3651 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:55.368708 3651 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.369261 3651 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:55.369663 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:55.369942 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:55.370383 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:55.370617 3651 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:55.712064 3651 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:55.712213 3651 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.712711 3651 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:55.714565 3572 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:55.714638 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:55.714697 3572 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.715103 3572 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:55.715499 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:55.715633 3361 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7594b0f72b47b796c0f3b8c608811f in term 1.
I20250114 03:03:55.715776 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:55.715893 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:55.716116 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f, 5a9be6271ae946edad7bd7193ae7419f; no voters: 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:55.716292 3651 raft_consensus.cc:2798] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 03:03:55.716348 3651 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:55.716392 3651 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:03:55.716485 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:55.716665 3572 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:03:55.717660 3651 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.718065 3651 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 election: Requested vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:55.718405 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f"
I20250114 03:03:55.718986 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e"
I20250114 03:03:55.719275 3229 raft_consensus.cc:1235] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 2 FOLLOWER]: Rejecting Update request from peer 2d76dc37a682421084593a4712c3be9e for earlier term 1. Current term is 2. Ops: []
I20250114 03:03:55.719364 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:55.719673 3651 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 03:03:55.719996 3661 consensus_queue.cc:1046] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 }, Status: INVALID_TERM, Last received: 1.43, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:03:55.720232 3653 raft_consensus.cc:3049] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:03:55.720293 3653 raft_consensus.cc:738] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d76dc37a682421084593a4712c3be9e, State: Running, Role: LEADER
I20250114 03:03:55.720458 3653 consensus_queue.cc:260] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 43, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.720749 3653 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:03:55.725747 3653 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:03:55.725878 3653 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.726341 3653 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995)
I20250114 03:03:55.726787 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 3 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:55.726787 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 3 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:55.726936 3229 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 2.
I20250114 03:03:55.726948 3360 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 1.
I20250114 03:03:55.727236 3032 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e; no voters:
I20250114 03:03:55.727396 3653 raft_consensus.cc:2798] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 03:03:55.727454 3653 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:03:55.727506 3653 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:03:55.728273 3653 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.728577 3653 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 3 election: Requested vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995)
I20250114 03:03:55.729063 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 3 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f"
I20250114 03:03:55.729074 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "2d76dc37a682421084593a4712c3be9e" candidate_term: 3 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f"
I20250114 03:03:55.729195 3360 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 1 FOLLOWER]: Advancing to term 3
I20250114 03:03:55.729195 3229 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:03:55.730188 3229 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 3.
I20250114 03:03:55.730188 3360 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d76dc37a682421084593a4712c3be9e in term 3.
I20250114 03:03:55.730508 3032 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f; no voters:
I20250114 03:03:55.730729 3653 raft_consensus.cc:2798] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:03:55.730832 3653 raft_consensus.cc:695] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [term 3 LEADER]: Becoming Leader. State: Replica: 2d76dc37a682421084593a4712c3be9e, State: Running, Role: LEADER
I20250114 03:03:55.731086 3653 consensus_queue.cc:237] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.43, Last appended by leader: 43, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:55.732764 2953 catalog_manager.cc:5526] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "2d76dc37a682421084593a4712c3be9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } health_report { overall_health: HEALTHY } } }
I20250114 03:03:55.733407 3229 raft_consensus.cc:1270] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Refusing update from remote peer 2d76dc37a682421084593a4712c3be9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 3 index: 44. (index mismatch)
I20250114 03:03:55.733521 3360 raft_consensus.cc:1270] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Refusing update from remote peer 2d76dc37a682421084593a4712c3be9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 43. Preceding OpId from leader: term: 3 index: 44. (index mismatch)
I20250114 03:03:55.733836 3653 consensus_queue.cc:1035] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:03:55.734066 3661 consensus_queue.cc:1035] T 758ade1cfcb14ffe9b6d10cff41b76df P 2d76dc37a682421084593a4712c3be9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
W20250114 03:03:55.852283 3635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60160: Aborted: Op submitted in term 1 cannot be replicated in term 3
I20250114 03:03:55.852813 3635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=42, attempt_no=0}) took 1555 ms. Trace:
I20250114 03:03:55.852921 3635 rpcz_store.cc:276] 0114 03:03:54.297166 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:54.297207 (+ 41us) service_pool.cc:225] Handling call
0114 03:03:55.852798 (+1555591us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:54.852669 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:54.852786 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:54.852791 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:54.852793 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:54.852902 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:54.852912 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:54.852914 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:54.852960 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:03:54.852961 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:54.852996 (+ 35us) write_op.cc:270] Start()
0114 03:03:55.852141 (+999145us) write_op.cc:276] Timestamp: P: 1736823834852949 usec, L: 0
0114 03:03:55.852145 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:55.852214 (+ 69us) op_driver.cc:389] HandleFailure(Aborted: Op submitted in term 1 cannot be replicated in term 3)
0114 03:03:55.852236 (+ 22us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:55.852249 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:55.852255 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":555154}]]}
W20250114 03:03:55.862485 3188 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56318: Illegal state: replica 0b7594b0f72b47b796c0f3b8c608811f is not leader of this config: current role FOLLOWER
W20250114 03:03:55.872395 3320 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40870: Illegal state: replica 5a9be6271ae946edad7bd7193ae7419f is not leader of this config: current role FOLLOWER
W20250114 03:03:56.513901 3635 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:60160: Aborted: Op submitted in term 1 cannot be replicated in term 3
I20250114 03:03:56.514108 3635 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=43, attempt_no=0}) took 1155 ms. Trace:
I20250114 03:03:56.514174 3635 rpcz_store.cc:276] 0114 03:03:55.358605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:55.358655 (+ 50us) service_pool.cc:225] Handling call
0114 03:03:56.514094 (+1155439us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:55.853366 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:55.853487 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:55.853491 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:03:55.853493 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:55.853565 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:55.853575 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:55.853578 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:55.853628 (+ 50us) tablet.cc:698] Row locks acquired
0114 03:03:55.853630 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:03:55.853663 (+ 33us) write_op.cc:270] Start()
0114 03:03:56.513808 (+660145us) write_op.cc:276] Timestamp: P: 1736823835853635 usec, L: 0
0114 03:03:56.513812 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:56.513846 (+ 34us) op_driver.cc:389] HandleFailure(Aborted: Op submitted in term 1 cannot be replicated in term 3)
0114 03:03:56.513869 (+ 23us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:56.513882 (+ 13us) write_op.cc:454] Released schema lock
0114 03:03:56.513888 (+ 6us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":494380}]]}
W20250114 03:03:56.517366 3188 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56318: Illegal state: replica 0b7594b0f72b47b796c0f3b8c608811f is not leader of this config: current role FOLLOWER
W20250114 03:03:56.521857 3320 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40870: Illegal state: replica 5a9be6271ae946edad7bd7193ae7419f is not leader of this config: current role FOLLOWER
W20250114 03:03:56.577076 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.694s user 0.001s sys 0.000s
W20250114 03:03:56.577854 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.694s user 0.000s sys 0.000s
I20250114 03:03:56.846755 3672 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:56.846923 3672 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:56.847453 3672 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:56.847939 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:56.848115 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:56.848711 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:56.849074 3672 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:03:56.937163 3673 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:56.937330 3673 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:56.937862 3673 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:56.938107 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:56.938632 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:56.939016 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:56.939188 3673 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:03:57.948951 3672 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:57.949105 3672 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:57.949627 3672 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:57.949993 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:57.950007 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:57.950518 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:57.950711 3672 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:03:57.952337 3686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=43, attempt_no=3}) took 1421 ms. Trace:
I20250114 03:03:57.952459 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 105) took 1223 ms. Trace:
I20250114 03:03:57.952459 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 104) took 1224 ms. Trace:
I20250114 03:03:57.952611 3055 rpcz_store.cc:276] 0114 03:03:56.728462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:56.728544 (+ 82us) service_pool.cc:225] Handling call
0114 03:03:56.728725 (+ 181us) tablet_service.cc:2780] Created scanner 6442908a06a44fdd809a2d6e14cd3754 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 1495c4312bec48ba81bb5636eb19eb95
0114 03:03:56.729020 (+ 295us) tablet_service.cc:2920] Creating iterator
0114 03:03:56.729047 (+ 27us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:56.988076 (+259029us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:57.951482 (+963406us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1222424 microseconds
0114 03:03:57.951571 (+ 89us) tablet_service.cc:2945] Iterator created
0114 03:03:57.951924 (+ 353us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:57.951952 (+ 28us) tablet_service.cc:3010] has_more: true
0114 03:03:57.951991 (+ 39us) tablet_service.cc:3027] Continuing scan request
0114 03:03:57.952050 (+ 59us) tablet_service.cc:3091] Found scanner 6442908a06a44fdd809a2d6e14cd3754 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 1495c4312bec48ba81bb5636eb19eb95
0114 03:03:57.952448 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":21888}
I20250114 03:03:57.952553 3056 rpcz_store.cc:276] 0114 03:03:56.728663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:56.728718 (+ 55us) service_pool.cc:225] Handling call
0114 03:03:56.728886 (+ 168us) tablet_service.cc:2780] Created scanner 1205ec1931964ce69f14232cb1dd83f1 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is bbcd9231055c44d78630fd40859ee76f
0114 03:03:56.729087 (+ 201us) tablet_service.cc:2920] Creating iterator
0114 03:03:56.729111 (+ 24us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:56.988801 (+259690us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:03:57.951482 (+962681us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1222360 microseconds
0114 03:03:57.951571 (+ 89us) tablet_service.cc:2945] Iterator created
0114 03:03:57.951924 (+ 353us) tablet_service.cc:2967] Iterator init: OK
0114 03:03:57.951955 (+ 31us) tablet_service.cc:3010] has_more: true
0114 03:03:57.951991 (+ 36us) tablet_service.cc:3027] Continuing scan request
0114 03:03:57.952050 (+ 59us) tablet_service.cc:3091] Found scanner 1205ec1931964ce69f14232cb1dd83f1 for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is bbcd9231055c44d78630fd40859ee76f
0114 03:03:57.952448 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":19456}
I20250114 03:03:57.952409 3686 rpcz_store.cc:276] 0114 03:03:56.531245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:56.531297 (+ 52us) service_pool.cc:225] Handling call
0114 03:03:57.952324 (+1421027us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:56.575471 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:56.575595 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:56.575601 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:56.575602 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:56.575702 (+ 100us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:56.575714 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:56.575718 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:56.575767 (+ 49us) tablet.cc:698] Row locks acquired
0114 03:03:56.575768 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:56.575808 (+ 40us) write_op.cc:270] Start()
0114 03:03:56.987960 (+412152us) write_op.cc:276] Timestamp: P: 1736823836575745 usec, L: 0
0114 03:03:56.987964 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:56.988214 (+ 250us) log.cc:844] Serialized 354 byte log entry
0114 03:03:57.950608 (+962394us) op_driver.cc:464] REPLICATION: finished
0114 03:03:57.951139 (+ 531us) write_op.cc:301] APPLY: starting
0114 03:03:57.951159 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:57.951193 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:57.951208 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:57.951274 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:57.951304 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:57.951309 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:03:57.951375 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:03:57.952241 (+ 866us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:57.952252 (+ 11us) write_op.cc:454] Released schema lock
0114 03:03:57.952257 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":159,"num_ops":1,"prepare.queue_time_us":43817,"prepare.run_cpu_time_us":922,"prepare.run_wall_time_us":412990,"raft.queue_time_us":96,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":962611,"spinlock_wait_cycles":95360,"thread_start_us":173,"threads_started":2}]]}
W20250114 03:03:57.953393 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.225s user 0.000s sys 0.000s
W20250114 03:03:57.953596 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.226s user 0.001s sys 0.000s
I20250114 03:03:57.972075 3673 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:57.972223 3673 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:57.972751 3673 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:57.973184 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:57.973420 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 46 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:57.974092 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:58.201617 3673 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:03:58.203111 3686 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=44, attempt_no=0}) took 1353 ms. Trace:
I20250114 03:03:58.203204 3686 rpcz_store.cc:276] 0114 03:03:56.849791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:56.849885 (+ 94us) service_pool.cc:225] Handling call
0114 03:03:58.203093 (+1353208us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:56.988485 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:56.988583 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:56.988588 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:56.988589 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:56.988666 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:56.988675 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:56.988678 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:56.988723 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:03:56.988724 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:56.988766 (+ 42us) write_op.cc:270] Start()
0114 03:03:57.393973 (+405207us) write_op.cc:276] Timestamp: P: 1736823836988714 usec, L: 0
0114 03:03:57.393978 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:03:57.394216 (+ 238us) log.cc:844] Serialized 354 byte log entry
0114 03:03:58.202557 (+808341us) op_driver.cc:464] REPLICATION: finished
0114 03:03:58.202676 (+ 119us) write_op.cc:301] APPLY: starting
0114 03:03:58.202697 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:58.202732 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:58.202752 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:58.202820 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:58.202854 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:58.202858 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:03:58.202933 (+ 75us) log.cc:844] Serialized 22 byte log entry
0114 03:03:58.202995 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:58.203009 (+ 14us) write_op.cc:454] Released schema lock
0114 03:03:58.203014 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"num_ops":1,"prepare.queue_time_us":138229,"prepare.run_cpu_time_us":713,"prepare.run_wall_time_us":405796,"replication_time_us":808543}]]}
I20250114 03:03:58.538090 3672 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:58.538218 3672 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:58.538698 3672 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:58.538969 3361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 47 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:58.539309 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 47 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:58.539662 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:58.539867 3672 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:03:58.768438 3672 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:58.768597 3672 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:58.769146 3672 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:58.769559 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:58.769694 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:58.770129 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:03:58.770888 3672 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250114 03:03:58.771625 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.668s user 0.001s sys 0.000s
W20250114 03:03:58.771675 3454 scanner-internal.cc:458] Time spent opening tablet: real 0.668s user 0.000s sys 0.000s
I20250114 03:03:58.980377 3673 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:58.980500 3673 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:58.981024 3673 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:58.981309 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:03:58.981400 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 48 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:58.981897 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:03:58.982128 3673 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:03:59.420092 3697 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=46, attempt_no=0}) took 1215 ms. Trace:
I20250114 03:03:59.420244 3697 rpcz_store.cc:276] 0114 03:03:58.204862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:58.204912 (+ 50us) service_pool.cc:225] Handling call
0114 03:03:59.420071 (+1215159us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:58.670145 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:58.670266 (+ 121us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:58.670271 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:03:58.670273 (+ 2us) tablet.cc:659] Decoding operations
0114 03:03:58.670355 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:58.670366 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:58.670369 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:58.670414 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:03:58.670415 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:58.670452 (+ 37us) write_op.cc:270] Start()
0114 03:03:59.416595 (+746143us) write_op.cc:276] Timestamp: P: 1736823838670381 usec, L: 0
0114 03:03:59.416599 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:03:59.416885 (+ 286us) log.cc:844] Serialized 354 byte log entry
0114 03:03:59.419473 (+ 2588us) op_driver.cc:464] REPLICATION: finished
0114 03:03:59.419727 (+ 254us) write_op.cc:301] APPLY: starting
0114 03:03:59.419743 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:03:59.419774 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:03:59.419788 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:03:59.419839 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:03:59.419863 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:03:59.419866 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:03:59.419912 (+ 46us) log.cc:844] Serialized 22 byte log entry
0114 03:03:59.419987 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:03:59.419997 (+ 10us) write_op.cc:454] Released schema lock
0114 03:03:59.420001 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"num_ops":1,"prepare.queue_time_us":464808,"prepare.run_cpu_time_us":836,"prepare.run_wall_time_us":746894,"raft.queue_time_us":34,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":18,"replication_time_us":2845,"spinlock_wait_cycles":147200,"thread_start_us":109,"threads_started":1}]]}
W20250114 03:03:59.422297 3453 scanner-internal.cc:458] Time spent opening tablet: real 0.500s user 0.000s sys 0.000s
I20250114 03:03:59.648468 3698 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:03:59.648636 3698 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:03:59.649191 3698 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:03:59.649634 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:03:59.649727 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 49 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:03:59.650154 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:04:00.091418 3699 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:04:00.091559 3699 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:04:00.092111 3699 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:04:00.092545 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 50 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:04:00.092540 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 50 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:04:00.093012 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:04:00.093429 3704 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=47, attempt_no=0}) took 1321 ms. Trace:
I20250114 03:04:00.093511 3704 rpcz_store.cc:276] 0114 03:03:58.771932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:58.771992 (+ 60us) service_pool.cc:225] Handling call
0114 03:04:00.093417 (+1321425us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:59.417070 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:59.417189 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:59.417196 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:03:59.417197 (+ 1us) tablet.cc:659] Decoding operations
0114 03:03:59.417272 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:59.417280 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:59.417282 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:59.417328 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:03:59.417329 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:59.417367 (+ 38us) write_op.cc:270] Start()
0114 03:03:59.477513 (+ 60146us) write_op.cc:276] Timestamp: P: 1736823839417335 usec, L: 0
0114 03:03:59.477516 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:03:59.477683 (+ 167us) log.cc:844] Serialized 354 byte log entry
0114 03:04:00.092502 (+614819us) op_driver.cc:464] REPLICATION: finished
0114 03:04:00.092891 (+ 389us) write_op.cc:301] APPLY: starting
0114 03:04:00.092909 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:00.092942 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:00.092955 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:00.093015 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:00.093047 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:00.093051 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:00.093118 (+ 67us) log.cc:844] Serialized 22 byte log entry
0114 03:04:00.093327 (+ 209us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:00.093340 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:00.093346 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":331,"num_ops":1,"prepare.queue_time_us":644725,"prepare.run_cpu_time_us":605,"prepare.run_wall_time_us":60676,"replication_time_us":614951,"thread_start_us":70,"threads_started":1}]]}
I20250114 03:04:00.535190 3698 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:04:00.905190 3699 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:04:00.906898 3710 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=48, attempt_no=0}) took 1485 ms. Trace:
I20250114 03:04:00.907066 3710 rpcz_store.cc:276] 0114 03:03:59.421563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:59.421620 (+ 57us) service_pool.cc:225] Handling call
0114 03:04:00.906871 (+1485251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:03:59.477782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:03:59.477897 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:03:59.477903 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:03:59.477906 (+ 3us) tablet.cc:659] Decoding operations
0114 03:03:59.477993 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:03:59.478003 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:03:59.478006 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:03:59.478051 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:03:59.478052 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:03:59.478088 (+ 36us) write_op.cc:270] Start()
0114 03:03:59.710217 (+232129us) write_op.cc:276] Timestamp: P: 1736823839478051 usec, L: 0
0114 03:03:59.710222 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:03:59.710539 (+ 317us) log.cc:844] Serialized 354 byte log entry
0114 03:04:00.906209 (+1195670us) op_driver.cc:464] REPLICATION: finished
0114 03:04:00.906477 (+ 268us) write_op.cc:301] APPLY: starting
0114 03:04:00.906497 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:00.906531 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:00.906547 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:00.906610 (+ 63us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:00.906633 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:00.906637 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:00.906694 (+ 57us) log.cc:844] Serialized 22 byte log entry
0114 03:04:00.906779 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:00.906792 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:00.906797 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":187,"num_ops":1,"prepare.queue_time_us":55854,"prepare.run_cpu_time_us":815,"prepare.run_wall_time_us":232851,"replication_time_us":1195955,"thread_start_us":84,"threads_started":1}]]}
I20250114 03:04:00.907845 3055 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 115) took 1335 ms. Trace:
I20250114 03:04:00.907933 3055 rpcz_store.cc:276] 0114 03:03:59.572582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:59.572684 (+ 102us) service_pool.cc:225] Handling call
0114 03:03:59.572900 (+ 216us) tablet_service.cc:2780] Created scanner 412616efa6384e928ce8b77b2426038f for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 4319fe4d44054540aa9e97649cbdfc05
0114 03:03:59.573122 (+ 222us) tablet_service.cc:2920] Creating iterator
0114 03:03:59.573159 (+ 37us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:59.710380 (+137221us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:04:00.906824 (+1196444us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1333648 microseconds
0114 03:04:00.906915 (+ 91us) tablet_service.cc:2945] Iterator created
0114 03:04:00.907274 (+ 359us) tablet_service.cc:2967] Iterator init: OK
0114 03:04:00.907296 (+ 22us) tablet_service.cc:3010] has_more: true
0114 03:04:00.907335 (+ 39us) tablet_service.cc:3027] Continuing scan request
0114 03:04:00.907387 (+ 52us) tablet_service.cc:3091] Found scanner 412616efa6384e928ce8b77b2426038f for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 4319fe4d44054540aa9e97649cbdfc05
0114 03:04:00.907839 (+ 452us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0,"spinlock_wait_cycles":30592}
I20250114 03:04:00.907840 3056 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:60160 (request call id 116) took 1334 ms. Trace:
W20250114 03:04:00.908375 3454 scanner-internal.cc:458] Time spent opening tablet: real 1.336s user 0.001s sys 0.000s
I20250114 03:04:00.908375 3056 rpcz_store.cc:276] 0114 03:03:59.573247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:03:59.573300 (+ 53us) service_pool.cc:225] Handling call
0114 03:03:59.573410 (+ 110us) tablet_service.cc:2780] Created scanner e215a977507547efb443086c6ecc477a for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 1a107053e0bd41dba75b56bbd9704756
0114 03:03:59.573544 (+ 134us) tablet_service.cc:2920] Creating iterator
0114 03:03:59.573562 (+ 18us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:03:59.710422 (+136860us) tablet_service.cc:3304] Waiting for operations to commit
0114 03:04:00.907074 (+1196652us) tablet_service.cc:3320] All operations in snapshot committed. Waited for 1333501 microseconds
0114 03:04:00.907149 (+ 75us) tablet_service.cc:2945] Iterator created
0114 03:04:00.907375 (+ 226us) tablet_service.cc:2967] Iterator init: OK
0114 03:04:00.907391 (+ 16us) tablet_service.cc:3010] has_more: true
0114 03:04:00.907415 (+ 24us) tablet_service.cc:3027] Continuing scan request
0114 03:04:00.907441 (+ 26us) tablet_service.cc:3091] Found scanner e215a977507547efb443086c6ecc477a for tablet 758ade1cfcb14ffe9b6d10cff41b76df, query id is 1a107053e0bd41dba75b56bbd9704756
0114 03:04:00.907827 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"rowset_iterators":1,"scanner_bytes_read":0}
W20250114 03:04:00.908811 3453 scanner-internal.cc:458] Time spent opening tablet: real 1.336s user 0.000s sys 0.000s
I20250114 03:04:01.400970 3698 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:04:01.401108 3698 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:04:01.401715 3698 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5a9be6271ae946edad7bd7193ae7419f (127.1.109.67:37995), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:04:01.402227 3360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "5a9be6271ae946edad7bd7193ae7419f" is_pre_election: true
I20250114 03:04:01.402225 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "0b7594b0f72b47b796c0f3b8c608811f" candidate_term: 4 candidate_status { last_received { term: 3 index: 51 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:04:01.402721 3164 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f; no voters: 2d76dc37a682421084593a4712c3be9e, 5a9be6271ae946edad7bd7193ae7419f
I20250114 03:04:01.445400 3699 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:04:01.445551 3699 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:04:01.446146 3699 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:04:01.446568 3229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:04:01.446576 3096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "2d76dc37a682421084593a4712c3be9e" is_pre_election: true
I20250114 03:04:01.447090 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:04:01.447125 3716 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:60160 (ReqId={client: e894bc4bb211496cb8b35e93db88a302, seq_no=49, attempt_no=0}) took 1352 ms. Trace:
I20250114 03:04:01.447189 3716 rpcz_store.cc:276] 0114 03:04:00.094641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:00.094716 (+ 75us) service_pool.cc:225] Handling call
0114 03:04:01.447111 (+1352395us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:00.095073 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 758ade1cfcb14ffe9b6d10cff41b76df
0114 03:04:00.095155 (+ 82us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:00.095158 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:00.095159 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:00.095210 (+ 51us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:00.095216 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:00.095217 (+ 1us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:00.095246 (+ 29us) tablet.cc:698] Row locks acquired
0114 03:04:00.095247 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:00.095271 (+ 24us) write_op.cc:270] Start()
0114 03:04:01.081446 (+986175us) write_op.cc:276] Timestamp: P: 1736823840095222 usec, L: 0
0114 03:04:01.081450 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:01.081639 (+ 189us) log.cc:844] Serialized 354 byte log entry
0114 03:04:01.446459 (+364820us) op_driver.cc:464] REPLICATION: finished
0114 03:04:01.446711 (+ 252us) write_op.cc:301] APPLY: starting
0114 03:04:01.446728 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:01.446776 (+ 48us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:01.446791 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:01.446848 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:01.446883 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:01.446888 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:01.446958 (+ 70us) log.cc:844] Serialized 22 byte log entry
0114 03:04:01.447018 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:01.447033 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:01.447038 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":176,"num_ops":1,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":573,"prepare.run_wall_time_us":986628,"replication_time_us":364978,"thread_start_us":77,"threads_started":1}]]}
I20250114 03:04:01.447398 3699 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:04:01.511495 3340 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:04:01.513727 3076 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:04:01.584602 3698 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250114 03:04:01.586524 3208 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
0ed628867e0c43889eff4cd01c13adec | 127.1.109.126:41883 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:40317 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0b7594b0f72b47b796c0f3b8c608811f | 127.1.109.66:35013 | HEALTHY | <none> | 0 | 0
2d76dc37a682421084593a4712c3be9e | 127.1.109.65:35251 | HEALTHY | <none> | 1 | 0
5a9be6271ae946edad7bd7193ae7419f | 127.1.109.67:37995 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:35251
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:35013
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:37995
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:35251
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:35013
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElections.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:37995
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:40317 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250114 03:04:01.609714 1461 log_verifier.cc:126] Checking tablet 758ade1cfcb14ffe9b6d10cff41b76df
I20250114 03:04:01.650597 1461 log_verifier.cc:177] Verified matching terms for 52 ops in tablet 758ade1cfcb14ffe9b6d10cff41b76df
I20250114 03:04:01.668795 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 3014
I20250114 03:04:01.674463 3699 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:04:01.674631 3699 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:04:01.675140 3699 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
I20250114 03:04:01.675465 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f" is_pre_election: true
I20250114 03:04:01.675594 3228 raft_consensus.cc:2463] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a9be6271ae946edad7bd7193ae7419f in term 3.
I20250114 03:04:01.675881 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7594b0f72b47b796c0f3b8c608811f, 5a9be6271ae946edad7bd7193ae7419f; no voters:
I20250114 03:04:01.676048 3699 raft_consensus.cc:2798] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 03:04:01.676105 3699 raft_consensus.cc:491] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Starting leader election (detected failure of leader 2d76dc37a682421084593a4712c3be9e)
I20250114 03:04:01.676137 3699 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 3 FOLLOWER]: Advancing to term 4
W20250114 03:04:01.676725 3296 connection.cc:537] client connection to 127.1.109.65:35251 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:04:01.676855 3296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20250114 03:04:01.677139 3699 raft_consensus.cc:513] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b7594b0f72b47b796c0f3b8c608811f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35013 } } peers { permanent_uuid: "5a9be6271ae946edad7bd7193ae7419f" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 37995 } } peers { permanent_uuid: "2d76dc37a682421084593a4712c3be9e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 35251 } }
I20250114 03:04:01.677497 3699 leader_election.cc:290] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 election: Requested vote from peers 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013), 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251)
W20250114 03:04:01.677656 3162 connection.cc:537] server connection from 127.1.109.65:45411 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 03:04:01.677924 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 3146
I20250114 03:04:01.677978 3228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758ade1cfcb14ffe9b6d10cff41b76df" candidate_uuid: "5a9be6271ae946edad7bd7193ae7419f" candidate_term: 4 candidate_status { last_received { term: 3 index: 52 } } ignore_live_leader: false dest_uuid: "0b7594b0f72b47b796c0f3b8c608811f"
W20250114 03:04:01.678018 3296 leader_election.cc:336] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251): Network error: Client connection negotiation failed: client connection to 127.1.109.65:35251: connect: Connection refused (error 111)
I20250114 03:04:01.678100 3228 raft_consensus.cc:3054] T 758ade1cfcb14ffe9b6d10cff41b76df P 0b7594b0f72b47b796c0f3b8c608811f [term 3 FOLLOWER]: Advancing to term 4
W20250114 03:04:01.678743 3296 leader_election.cc:336] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 2d76dc37a682421084593a4712c3be9e (127.1.109.65:35251): Network error: Client connection negotiation failed: client connection to 127.1.109.65:35251: connect: Connection refused (error 111)
W20250114 03:04:01.685154 3753 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:04:01.684736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.109.66:35013 (local address 127.1.109.67:53083)
0114 03:04:01.685010 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0114 03:04:01.685054 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.1.109.66:35013: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":164}
W20250114 03:04:01.685292 3296 leader_election.cc:336] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 0b7594b0f72b47b796c0f3b8c608811f (127.1.109.66:35013): Network error: Client connection negotiation failed: client connection to 127.1.109.66:35013: connect: Connection reset by peer (error 104)
I20250114 03:04:01.685369 3296 leader_election.cc:304] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a9be6271ae946edad7bd7193ae7419f; no voters: 0b7594b0f72b47b796c0f3b8c608811f, 2d76dc37a682421084593a4712c3be9e
I20250114 03:04:01.685472 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 3278
I20250114 03:04:01.685588 3699 raft_consensus.cc:2743] T 758ade1cfcb14ffe9b6d10cff41b76df P 5a9be6271ae946edad7bd7193ae7419f [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250114 03:04:01.691243 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 2923
2025-01-14T03:04:01Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElections (32409 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency
I20250114 03:04:01.716542 1461 test_util.cc:274] Using random seed: -803243720
I20250114 03:04:01.717370 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:04:01.717417 1461 ts_itest-base.cc:113] --------------
I20250114 03:04:01.717427 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:04:01.717438 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:04:01.717451 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:04:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:04:01Z Disabled control of system clock
I20250114 03:04:01.725625 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34647
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:34647 with env {}
W20250114 03:04:01.835048 3761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:01.835280 3761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:01.835321 3761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:01.838455 3761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:04:01.838521 3761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:01.838541 3761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:04:01.838559 3761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:04:01.842393 3761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:34647
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.3761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:01.843515 3761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:01.844796 3761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:01.853113 3766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:01.853896 3761 server_base.cc:1034] running on GCE node
W20250114 03:04:01.854755 3767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:01.855528 3769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:01.856104 3761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:01.857204 3761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:01.858376 3761 hybrid_clock.cc:648] HybridClock initialized: now 1736823841858348 us; error 42 us; skew 500 ppm
I20250114 03:04:01.860586 3761 webserver.cc:458] Webserver started at http://127.1.109.126:44047/ using document root <none> and password file <none>
I20250114 03:04:01.861233 3761 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:01.861295 3761 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:01.861469 3761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:01.863176 3761 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bcc99ceacac04f819c6bc0b05b700fa3"
format_stamp: "Formatted at 2025-01-14 03:04:01 on dist-test-slave-r16g"
I20250114 03:04:01.863662 3761 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bcc99ceacac04f819c6bc0b05b700fa3"
format_stamp: "Formatted at 2025-01-14 03:04:01 on dist-test-slave-r16g"
I20250114 03:04:01.867230 3761 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:04:01.869475 3775 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:01.870525 3761 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:04:01.870640 3761 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bcc99ceacac04f819c6bc0b05b700fa3"
format_stamp: "Formatted at 2025-01-14 03:04:01 on dist-test-slave-r16g"
I20250114 03:04:01.870765 3761 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:01.914862 3761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:01.915560 3761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:01.915740 3761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:01.922683 3761 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:34647
I20250114 03:04:01.922708 3828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:34647 every 8 connection(s)
I20250114 03:04:01.923673 3761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:04:01.926362 3829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:01.931344 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 3761
I20250114 03:04:01.931443 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:04:01.931591 3829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3: Bootstrap starting.
I20250114 03:04:01.934137 3829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:01.934933 3829 log.cc:826] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:01.936863 3829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3: No bootstrap required, opened a new log
I20250114 03:04:01.940248 3829 raft_consensus.cc:357] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34647 } }
I20250114 03:04:01.940446 3829 raft_consensus.cc:383] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:01.940482 3829 raft_consensus.cc:738] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcc99ceacac04f819c6bc0b05b700fa3, State: Initialized, Role: FOLLOWER
I20250114 03:04:01.940950 3829 consensus_queue.cc:260] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34647 } }
I20250114 03:04:01.941115 3829 raft_consensus.cc:397] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:04:01.941181 3829 raft_consensus.cc:491] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:04:01.941287 3829 raft_consensus.cc:3054] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:01.942094 3829 raft_consensus.cc:513] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34647 } }
I20250114 03:04:01.942412 3829 leader_election.cc:304] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bcc99ceacac04f819c6bc0b05b700fa3; no voters:
I20250114 03:04:01.942691 3829 leader_election.cc:290] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:04:01.942813 3834 raft_consensus.cc:2798] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:04:01.943034 3834 raft_consensus.cc:695] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [term 1 LEADER]: Becoming Leader. State: Replica: bcc99ceacac04f819c6bc0b05b700fa3, State: Running, Role: LEADER
I20250114 03:04:01.943332 3834 consensus_queue.cc:237] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34647 } }
I20250114 03:04:01.943930 3829 sys_catalog.cc:564] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:04:01.945101 3835 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34647 } } }
I20250114 03:04:01.945225 3835 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [sys.catalog]: This master's current role is: LEADER
I20250114 03:04:01.945472 3836 sys_catalog.cc:455] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bcc99ceacac04f819c6bc0b05b700fa3. Latest consensus state: current_term: 1 leader_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bcc99ceacac04f819c6bc0b05b700fa3" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34647 } } }
I20250114 03:04:01.945559 3836 sys_catalog.cc:458] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3 [sys.catalog]: This master's current role is: LEADER
I20250114 03:04:01.945673 3843 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:04:01.948796 3843 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:04:01.958391 3843 catalog_manager.cc:1348] Generated new cluster ID: a32407c53a2a4fa3b33beeb0feef51ba
I20250114 03:04:01.958488 3843 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:04:01.985981 3843 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:04:01.986994 3843 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:04:01.996420 3843 catalog_manager.cc:5899] T 00000000000000000000000000000000 P bcc99ceacac04f819c6bc0b05b700fa3: Generated new TSK 0
I20250114 03:04:01.997210 3843 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:04:02.007725 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34647
--builtin_ntp_servers=127.1.109.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
W20250114 03:04:02.122797 3853 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20250114 03:04:02.123091 3853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:02.123127 3853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:02.123185 3853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:02.126365 3853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:02.126492 3853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:04:02.130172 3853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:34647
--never_fsync=true
--heap_profile_path=/tmp/kudu.3853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:02.131258 3853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:02.132346 3853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:02.141448 3861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:02.142095 3858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:02.143098 3859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:02.143921 3853 server_base.cc:1034] running on GCE node
I20250114 03:04:02.144317 3853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:02.144943 3853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:02.146104 3853 hybrid_clock.cc:648] HybridClock initialized: now 1736823842146068 us; error 44 us; skew 500 ppm
I20250114 03:04:02.148101 3853 webserver.cc:458] Webserver started at http://127.1.109.65:46807/ using document root <none> and password file <none>
I20250114 03:04:02.148661 3853 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:02.148720 3853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:02.148924 3853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:02.152165 3853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3a063722f0224c6bbbcfb6b63ce83420"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.152633 3853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3a063722f0224c6bbbcfb6b63ce83420"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.156195 3853 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250114 03:04:02.158300 3867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.159252 3853 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:04:02.159368 3853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3a063722f0224c6bbbcfb6b63ce83420"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.159453 3853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:02.186790 3853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:02.187446 3853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:02.187618 3853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:02.188180 3853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:04:02.189124 3853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:04:02.189174 3853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.189231 3853 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:04:02.189262 3853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.198552 3853 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:40781
I20250114 03:04:02.198572 3981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:40781 every 8 connection(s)
I20250114 03:04:02.199523 3853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:04:02.204943 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 3853
I20250114 03:04:02.205068 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:04:02.207525 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34647
--builtin_ntp_servers=127.1.109.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20250114 03:04:02.208561 3982 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34647
I20250114 03:04:02.208863 3982 heartbeater.cc:463] Registering TS with master...
I20250114 03:04:02.209475 3982 heartbeater.cc:510] Master 127.1.109.126:34647 requested a full tablet report, sending...
I20250114 03:04:02.211176 3791 ts_manager.cc:194] Registered new tserver with Master: 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781)
I20250114 03:04:02.213029 3791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:60709
W20250114 03:04:02.317024 3985 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20250114 03:04:02.317279 3985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:02.317301 3985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:02.317384 3985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:02.320416 3985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:02.320523 3985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:04:02.324242 3985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:34647
--never_fsync=true
--heap_profile_path=/tmp/kudu.3985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:02.325325 3985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:02.326404 3985 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:02.335381 3990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:02.336045 3991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:02.337055 3993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:02.337880 3985 server_base.cc:1034] running on GCE node
I20250114 03:04:02.338299 3985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:02.338900 3985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:02.340047 3985 hybrid_clock.cc:648] HybridClock initialized: now 1736823842340024 us; error 39 us; skew 500 ppm
I20250114 03:04:02.342188 3985 webserver.cc:458] Webserver started at http://127.1.109.66:46283/ using document root <none> and password file <none>
I20250114 03:04:02.342770 3985 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:02.342824 3985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:02.343027 3985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:02.344695 3985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8ac0078d577b48d8b07432520bba2e4f"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.345242 3985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8ac0078d577b48d8b07432520bba2e4f"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.348904 3985 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250114 03:04:02.351096 3999 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.352128 3985 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:04:02.352228 3985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8ac0078d577b48d8b07432520bba2e4f"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.352342 3985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:02.369413 3985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:02.370011 3985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:02.370169 3985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:02.370695 3985 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:04:02.371613 3985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:04:02.371672 3985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.371714 3985 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:04:02.371754 3985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.380353 3985 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:35297
I20250114 03:04:02.380373 4113 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:35297 every 8 connection(s)
I20250114 03:04:02.381366 3985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:04:02.383623 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 3985
I20250114 03:04:02.383719 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:04:02.385984 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34647
--builtin_ntp_servers=127.1.109.84:37681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000
--consensus_inject_latency_ms_in_notifications=50 with env {}
I20250114 03:04:02.390655 4114 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34647
I20250114 03:04:02.390893 4114 heartbeater.cc:463] Registering TS with master...
I20250114 03:04:02.391407 4114 heartbeater.cc:510] Master 127.1.109.126:34647 requested a full tablet report, sending...
I20250114 03:04:02.392426 3791 ts_manager.cc:194] Registered new tserver with Master: 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:02.393002 3791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:50007
W20250114 03:04:02.499665 4117 flags.cc:425] Enabled unsafe flag: --consensus_inject_latency_ms_in_notifications=50
W20250114 03:04:02.499946 4117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:02.499994 4117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:02.500078 4117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:02.503357 4117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:02.503535 4117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:04:02.507440 4117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_inject_latency_ms_in_notifications=50
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:34647
--never_fsync=true
--heap_profile_path=/tmp/kudu.4117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:02.508553 4117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:02.509791 4117 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:02.519042 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:02.519564 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:02.520490 4125 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:02.521450 4117 server_base.cc:1034] running on GCE node
I20250114 03:04:02.521864 4117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:02.522452 4117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:02.523609 4117 hybrid_clock.cc:648] HybridClock initialized: now 1736823842523585 us; error 39 us; skew 500 ppm
I20250114 03:04:02.525732 4117 webserver.cc:458] Webserver started at http://127.1.109.67:45095/ using document root <none> and password file <none>
I20250114 03:04:02.526310 4117 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:02.526374 4117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:02.526533 4117 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:02.528225 4117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e1f21921611b433b9bbf44c95152b278"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.528689 4117 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e1f21921611b433b9bbf44c95152b278"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.532344 4117 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250114 03:04:02.534716 4131 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.535779 4117 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:04:02.535879 4117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e1f21921611b433b9bbf44c95152b278"
format_stamp: "Formatted at 2025-01-14 03:04:02 on dist-test-slave-r16g"
I20250114 03:04:02.535962 4117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:02.557844 4117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:02.558441 4117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:02.558589 4117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:02.559111 4117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:04:02.560021 4117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:04:02.560067 4117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.560107 4117 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:04:02.560127 4117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:02.569195 4117 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:39007
I20250114 03:04:02.569242 4245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:39007 every 8 connection(s)
I20250114 03:04:02.570170 4117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:04:02.572157 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 4117
I20250114 03:04:02.572268 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:04:02.579655 4246 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34647
I20250114 03:04:02.579916 4246 heartbeater.cc:463] Registering TS with master...
I20250114 03:04:02.580497 4246 heartbeater.cc:510] Master 127.1.109.126:34647 requested a full tablet report, sending...
I20250114 03:04:02.581740 3791 ts_manager.cc:194] Registered new tserver with Master: e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:02.582304 3791 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:55511
I20250114 03:04:02.586166 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:04:02.587565 1461 test_util.cc:274] Using random seed: -802372696
I20250114 03:04:02.596840 3791 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48910:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 03:04:02.598047 3791 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:04:02.613886 4047 tablet_service.cc:1467] Processing CreateTablet for tablet d5bcdea4a70943628babc9dfd8c94f1e (DEFAULT_TABLE table=test-workload [id=0d0c42f365e64bc1bd0760b81d32cf5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:04:02.613924 4179 tablet_service.cc:1467] Processing CreateTablet for tablet d5bcdea4a70943628babc9dfd8c94f1e (DEFAULT_TABLE table=test-workload [id=0d0c42f365e64bc1bd0760b81d32cf5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:04:02.613919 3903 tablet_service.cc:1467] Processing CreateTablet for tablet d5bcdea4a70943628babc9dfd8c94f1e (DEFAULT_TABLE table=test-workload [id=0d0c42f365e64bc1bd0760b81d32cf5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:04:02.614926 4179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bcdea4a70943628babc9dfd8c94f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:02.614918 3903 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bcdea4a70943628babc9dfd8c94f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:02.614918 4047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5bcdea4a70943628babc9dfd8c94f1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:02.620469 4263 tablet_bootstrap.cc:492] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: Bootstrap starting.
I20250114 03:04:02.621711 4265 tablet_bootstrap.cc:492] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278: Bootstrap starting.
I20250114 03:04:02.622635 4263 tablet_bootstrap.cc:654] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:02.623371 4263 log.cc:826] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:02.623421 4265 tablet_bootstrap.cc:654] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:02.624131 4265 log.cc:826] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:02.624413 4264 tablet_bootstrap.cc:492] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: Bootstrap starting.
I20250114 03:04:02.625316 4263 tablet_bootstrap.cc:492] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: No bootstrap required, opened a new log
I20250114 03:04:02.625510 4263 ts_tablet_manager.cc:1397] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:04:02.626005 4265 tablet_bootstrap.cc:492] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278: No bootstrap required, opened a new log
I20250114 03:04:02.626169 4265 ts_tablet_manager.cc:1397] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:04:02.626348 4264 tablet_bootstrap.cc:654] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:02.626958 4264 log.cc:826] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:02.628343 4264 tablet_bootstrap.cc:492] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: No bootstrap required, opened a new log
I20250114 03:04:02.628510 4264 ts_tablet_manager.cc:1397] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250114 03:04:02.629619 4263 raft_consensus.cc:357] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.630009 4263 raft_consensus.cc:383] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:02.629952 4265 raft_consensus.cc:357] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.630110 4263 raft_consensus.cc:738] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac0078d577b48d8b07432520bba2e4f, State: Initialized, Role: FOLLOWER
I20250114 03:04:02.630224 4265 raft_consensus.cc:383] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:02.630326 4265 raft_consensus.cc:738] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1f21921611b433b9bbf44c95152b278, State: Initialized, Role: FOLLOWER
I20250114 03:04:02.630784 4263 consensus_queue.cc:260] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.631006 4265 consensus_queue.cc:260] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.631310 4270 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:02.631515 4270 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.631845 4263 ts_tablet_manager.cc:1428] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 03:04:02.631862 4265 ts_tablet_manager.cc:1428] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20250114 03:04:02.632184 4115 tablet.cc:2367] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:04:02.632272 4246 heartbeater.cc:502] Master 127.1.109.126:34647 was elected leader, sending a full tablet report...
I20250114 03:04:02.632272 4114 heartbeater.cc:502] Master 127.1.109.126:34647 was elected leader, sending a full tablet report...
I20250114 03:04:02.632228 4264 raft_consensus.cc:357] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.632417 4264 raft_consensus.cc:383] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:02.632470 4264 raft_consensus.cc:738] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a063722f0224c6bbbcfb6b63ce83420, State: Initialized, Role: FOLLOWER
I20250114 03:04:02.632969 4264 consensus_queue.cc:260] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.633913 4264 ts_tablet_manager.cc:1428] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250114 03:04:02.634534 4270 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:02.634912 4269 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:02.634920 4273 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:02.635064 4273 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.635159 4269 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.635643 3982 heartbeater.cc:502] Master 127.1.109.126:34647 was elected leader, sending a full tablet report...
I20250114 03:04:02.637326 4269 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:02.638415 4273 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:02.640869 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
W20250114 03:04:02.641053 4132 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.65:40781 timed out after 0.007s (SENT)
I20250114 03:04:02.641080 3935 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f21921611b433b9bbf44c95152b278 in term 0.
I20250114 03:04:02.642374 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "3a063722f0224c6bbbcfb6b63ce83420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:02.642557 4064 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a063722f0224c6bbbcfb6b63ce83420 in term 0.
I20250114 03:04:02.643247 3869 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f; no voters:
W20250114 03:04:02.643733 4000 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.65:40781 timed out after 0.007s (ON_OUTBOUND_QUEUE)
I20250114 03:04:02.643944 4274 raft_consensus.cc:2798] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:04:02.644013 4274 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:02.644066 4274 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:02.645280 4274 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.645412 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:02.645598 3935 raft_consensus.cc:2388] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ac0078d577b48d8b07432520bba2e4f in current term 1: Already voted for candidate 3a063722f0224c6bbbcfb6b63ce83420 in this term.
W20250114 03:04:02.645989 3871 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.67:39007 timed out after 0.008s (SENT)
I20250114 03:04:02.645994 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "3a063722f0224c6bbbcfb6b63ce83420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f"
I20250114 03:04:02.646106 4064 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:02.646896 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:02.647104 4199 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ac0078d577b48d8b07432520bba2e4f in term 0.
I20250114 03:04:02.647241 4064 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a063722f0224c6bbbcfb6b63ce83420 in term 1.
I20250114 03:04:02.647419 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "3a063722f0224c6bbbcfb6b63ce83420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:02.647539 4198 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a063722f0224c6bbbcfb6b63ce83420 in term 0.
I20250114 03:04:02.647521 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f, e1f21921611b433b9bbf44c95152b278; no voters:
I20250114 03:04:02.647564 3869 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f; no voters:
I20250114 03:04:02.647747 4273 raft_consensus.cc:2798] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:04:02.647938 4273 raft_consensus.cc:695] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 LEADER]: Becoming Leader. State: Replica: 3a063722f0224c6bbbcfb6b63ce83420, State: Running, Role: LEADER
I20250114 03:04:02.647976 4271 raft_consensus.cc:2758] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250114 03:04:02.648277 4273 consensus_queue.cc:237] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.648993 4274 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [CANDIDATE]: Term 1 election: Requested vote from peers 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:02.649315 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "3a063722f0224c6bbbcfb6b63ce83420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278"
I20250114 03:04:02.649415 4198 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:02.649693 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:02.649915 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "3a063722f0224c6bbbcfb6b63ce83420" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:02.650557 4198 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a063722f0224c6bbbcfb6b63ce83420 in term 1.
I20250114 03:04:02.652230 4274 consensus_queue.cc:1035] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:04:02.651971 3790 catalog_manager.cc:5526] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "3a063722f0224c6bbbcfb6b63ce83420" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } health_report { overall_health: UNKNOWN } } }
I20250114 03:04:02.652763 4273 consensus_queue.cc:1035] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:04:02.652769 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:02.654623 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:02.655076 4270 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250114 03:04:02.656324 4273 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:04:02.656368 4273 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250114 03:04:02.700572 3983 tablet.cc:2367] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:04:02.824555 4158 tablet_service.cc:2970] error setting up scanner 8e8cff60d1114f9896f35d22801e7940 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.147s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:02.824685 4159 tablet_service.cc:2970] error setting up scanner 73f76d1604364c92a9c0897f22835b66 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.147s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
W20250114 03:04:02.841502 4159 tablet_service.cc:2970] error setting up scanner 14c0379467c945d88d8347a03ae755c7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.164s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:02.844163 4159 tablet_service.cc:2970] error setting up scanner d32b34ccdf714bd0a396928ee706004f with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.167s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
W20250114 03:04:02.846776 4198 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:04:02.846839 4198 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250114 03:04:02.847716 4067 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:04:02.847771 4067 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250114 03:04:02.860357 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:02.860512 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:02.861302 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:02.861891 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:02.861966 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:02.862437 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:02.862691 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:02.863297 4159 tablet_service.cc:2970] error setting up scanner fd59811f78ba4819ad6d94571fe1e270 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.186s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:02.865788 4159 tablet_service.cc:2970] error setting up scanner ed8655cd6f6243478d7add9ed3d97d22 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.188s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
W20250114 03:04:02.913134 4159 tablet_service.cc:2970] error setting up scanner d89710f1bf7d43dabb56e632dec901b7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.235s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:02.915467 4159 tablet_service.cc:2970] error setting up scanner e6c32e9a1c264cb6a300886d8f0854e1 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.238s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
W20250114 03:04:03.019039 4159 tablet_service.cc:2970] error setting up scanner a5d4aa95dad7432fa46c62735978e2b9 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.341s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:03.069370 4159 tablet_service.cc:2970] error setting up scanner 4265464d0f8f482f9e6ecefa171fdc0a with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.392s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
W20250114 03:04:03.196939 4159 tablet_service.cc:2970] error setting up scanner cc6e25c867e44288aaf36b93ffe10bf8 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.519s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:03.210078 4295 mvcc.cc:204] Tried to move back new op lower bound from 7114030459605192704 to 7114030459490578432. Current Snapshot: MvccSnapshot[applied={T|T < 7114030459605192704 or (T in {7114030459605192704})}]
W20250114 03:04:03.263334 4159 tablet_service.cc:2970] error setting up scanner 32efa9d3424b497481b1043891470c51 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.586s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:03.776207 4270 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:03.776355 4270 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:03.777136 4270 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:03.777634 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:03.777653 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:03.778152 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:03.953737 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:03.953914 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:03.954654 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:03.954936 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:03.955091 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:03.955559 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:03.975932 4306 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=0, attempt_no=0}) took 1302 ms. Trace:
I20250114 03:04:03.976047 4306 rpcz_store.cc:276] 0114 03:04:02.673162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:02.673222 (+ 60us) service_pool.cc:225] Handling call
0114 03:04:03.975910 (+1302688us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:02.845324 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:02.845421 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:02.845427 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:02.845429 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:02.845504 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:02.845534 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:02.845537 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:02.845601 (+ 64us) tablet.cc:698] Row locks acquired
0114 03:04:02.845603 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:02.845641 (+ 38us) write_op.cc:270] Start()
0114 03:04:03.209779 (+364138us) write_op.cc:276] Timestamp: P: 1736823842845344 usec, L: 0
0114 03:04:03.209783 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:03.209950 (+ 167us) log.cc:844] Serialized 345 byte log entry
0114 03:04:03.975077 (+765127us) op_driver.cc:464] REPLICATION: finished
0114 03:04:03.975392 (+ 315us) write_op.cc:301] APPLY: starting
0114 03:04:03.975413 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:03.975455 (+ 42us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:03.975484 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:03.975578 (+ 94us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:03.975606 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:03.975609 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:03.975720 (+ 111us) log.cc:844] Serialized 22 byte log entry
0114 03:04:03.975804 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:03.975818 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:03.975822 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":231,"num_ops":1,"prepare.queue_time_us":169307,"prepare.run_cpu_time_us":654,"prepare.run_wall_time_us":364708,"replication_time_us":765239,"thread_start_us":103,"threads_started":1}]]}
I20250114 03:04:04.450625 4270 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:04.464221 4306 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=2, attempt_no=0}) took 1594 ms. Trace:
I20250114 03:04:04.464311 4306 rpcz_store.cc:276] 0114 03:04:02.869762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:02.869815 (+ 53us) service_pool.cc:225] Handling call
0114 03:04:04.464204 (+1594389us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:03.210274 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:03.210389 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:03.210394 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:03.210396 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:03.210477 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:03.210486 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:03.210489 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:03.210553 (+ 64us) tablet.cc:698] Row locks acquired
0114 03:04:03.210555 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:03.210597 (+ 42us) write_op.cc:270] Start()
0114 03:04:03.318735 (+108138us) write_op.cc:276] Timestamp: P: 1736823843210560 usec, L: 0
0114 03:04:03.318738 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:03.318957 (+ 219us) log.cc:844] Serialized 354 byte log entry
0114 03:04:04.463720 (+1144763us) op_driver.cc:464] REPLICATION: finished
0114 03:04:04.463829 (+ 109us) write_op.cc:301] APPLY: starting
0114 03:04:04.463851 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:04.463882 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:04.463898 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:04.463958 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:04.463996 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:04.464000 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:04.464067 (+ 67us) log.cc:844] Serialized 22 byte log entry
0114 03:04:04.464114 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:04.464128 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:04.464132 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":339596,"prepare.run_cpu_time_us":684,"prepare.run_wall_time_us":108746,"replication_time_us":1144926}]]}
W20250114 03:04:04.807320 4159 tablet_service.cc:2970] error setting up scanner 4ecf8805c542402f99067f8ddecddc54 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823843807037 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823843210560 usec, L: 0 Physical time difference: 0.596s: new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:04.807694 4159 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:52220 (request call id 12) took 1001 ms. Trace:
I20250114 03:04:04.807771 4159 rpcz_store.cc:276] 0114 03:04:03.806514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:03.806589 (+ 75us) service_pool.cc:225] Handling call
0114 03:04:03.806782 (+ 193us) tablet_service.cc:2780] Created scanner 4ecf8805c542402f99067f8ddecddc54 for tablet d5bcdea4a70943628babc9dfd8c94f1e, query id is ee819643cb794686999f6fe006fdff21
0114 03:04:03.807057 (+ 275us) tablet_service.cc:2920] Creating iterator
0114 03:04:03.807083 (+ 26us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:04:04.807294 (+1000211us) tablet_service.cc:2945] Iterator created
0114 03:04:04.807308 (+ 14us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823843807037 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823843210560 usec, L: 0 Physical time difference: 0.596s
0114 03:04:04.807681 (+ 373us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 03:04:04.820550 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:04.873406 4158 tablet_service.cc:2970] error setting up scanner 74def89772654b78853aaf9f70431b08 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823843873143 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823843210560 usec, L: 0 Physical time difference: 0.663s: new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:04.873853 4158 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:52220 (request call id 13) took 1001 ms. Trace:
I20250114 03:04:04.873934 4158 rpcz_store.cc:276] 0114 03:04:03.872667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:03.872744 (+ 77us) service_pool.cc:225] Handling call
0114 03:04:03.872958 (+ 214us) tablet_service.cc:2780] Created scanner 74def89772654b78853aaf9f70431b08 for tablet d5bcdea4a70943628babc9dfd8c94f1e, query id is f29e5d225b314cab9957bc0e7a2d8e4e
0114 03:04:03.873169 (+ 211us) tablet_service.cc:2920] Creating iterator
0114 03:04:03.873195 (+ 26us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:04:04.873375 (+1000180us) tablet_service.cc:2945] Iterator created
0114 03:04:04.873393 (+ 18us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823843873143 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823843210560 usec, L: 0 Physical time difference: 0.663s
0114 03:04:04.873836 (+ 443us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 03:04:05.001394 4270 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:05.001552 4270 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:05.002072 4270 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:05.002471 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:05.002885 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:05.003368 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:05.015311 4315 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=3, attempt_no=0}) took 1037 ms. Trace:
I20250114 03:04:05.015404 4315 rpcz_store.cc:276] 0114 03:04:03.977851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:03.977908 (+ 57us) service_pool.cc:225] Handling call
0114 03:04:05.015286 (+1037378us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:03.978440 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:03.978528 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:03.978531 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:03.978532 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:03.978589 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:03.978595 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:03.978596 (+ 1us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:03.978626 (+ 30us) tablet.cc:698] Row locks acquired
0114 03:04:03.978627 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:03.978655 (+ 28us) write_op.cc:270] Start()
0114 03:04:04.360942 (+382287us) write_op.cc:276] Timestamp: P: 1736823843978581 usec, L: 0
0114 03:04:04.360946 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:04.361170 (+ 224us) log.cc:844] Serialized 354 byte log entry
0114 03:04:05.014614 (+653444us) op_driver.cc:464] REPLICATION: finished
0114 03:04:05.014886 (+ 272us) write_op.cc:301] APPLY: starting
0114 03:04:05.014905 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:05.014943 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:05.014959 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:05.015018 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:05.015041 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:05.015044 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:05.015108 (+ 64us) log.cc:844] Serialized 22 byte log entry
0114 03:04:05.015176 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:05.015191 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:05.015196 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":211,"num_ops":1,"prepare.queue_time_us":148,"prepare.run_cpu_time_us":615,"prepare.run_wall_time_us":382796,"replication_time_us":653620,"thread_start_us":169,"threads_started":2}]]}
I20250114 03:04:05.192970 4270 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:05.632351 4270 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:05.632505 4270 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:05.633085 4270 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:05.633525 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:05.633558 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:05.633985 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:05.634207 4270 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:05.718079 4270 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:05.718235 4270 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:05.718768 4270 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:05.719188 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:05.719214 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:05.719630 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:05.719811 4270 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:06.231900 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:06.232051 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:06.232328 4067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:59569 (request call id 466) took 1410 ms. Trace:
I20250114 03:04:06.232537 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:06.233018 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:06.233080 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:06.232429 4067 rpcz_store.cc:276] 0114 03:04:04.821919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:04.821992 (+ 73us) service_pool.cc:225] Handling call
0114 03:04:04.822140 (+ 148us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:04.822215 (+ 75us) raft_consensus.cc:1529] Early marking committed up to index 4
0114 03:04:04.822357 (+ 142us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:04.822477 (+ 120us) write_op.cc:270] Start()
0114 03:04:05.575625 (+753148us) write_op.cc:276] Timestamp: P: 1736823843978581 usec, L: 0
0114 03:04:05.576049 (+ 424us) write_op.cc:270] Start()
0114 03:04:06.231236 (+655187us) write_op.cc:276] Timestamp: P: 1736823844466229 usec, L: 0
0114 03:04:06.231693 (+ 457us) log.cc:844] Serialized 708 byte log entry
0114 03:04:06.231832 (+ 139us) raft_consensus.cc:1640] Marking committed up to 4
0114 03:04:06.231838 (+ 6us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:06.231865 (+ 27us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:06.232265 (+ 400us) raft_consensus.cc:1680] finished
0114 03:04:06.232267 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:06.232311 (+ 44us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:05.575972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:05.576110 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:05.576114 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:05.576115 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:05.576181 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:05.576190 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:05.576192 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:05.576229 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:04:05.576230 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:04:06.231575 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:06.231710 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:06.231714 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:06.231716 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:06.231798 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:06.231836 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:06.231840 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:06.231893 (+ 53us) tablet.cc:698] Row locks acquired
0114 03:04:06.231895 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":76,"threads_started":1,"wal-append.queue_time_us":138,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":193,"prepare.run_cpu_time_us":404,"prepare.run_wall_time_us":404,"thread_start_us":81,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":31,"prepare.queue_time_us":198,"prepare.run_cpu_time_us":412,"prepare.run_wall_time_us":411,"thread_start_us":86,"threads_started":1}]]}
I20250114 03:04:06.233461 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:06.233685 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:06.706485 4158 tablet_service.cc:2970] error setting up scanner b5a8f2fd05e64336a281f09c163304ab with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823845706225 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823845208315 usec, L: 0 Physical time difference: 0.498s: new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:06.706888 4158 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:52220 (request call id 14) took 1001 ms. Trace:
I20250114 03:04:06.706971 4158 rpcz_store.cc:276] 0114 03:04:05.705607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:05.705724 (+ 117us) service_pool.cc:225] Handling call
0114 03:04:05.705972 (+ 248us) tablet_service.cc:2780] Created scanner b5a8f2fd05e64336a281f09c163304ab for tablet d5bcdea4a70943628babc9dfd8c94f1e, query id is ee819643cb794686999f6fe006fdff21
0114 03:04:05.706228 (+ 256us) tablet_service.cc:2920] Creating iterator
0114 03:04:05.706257 (+ 29us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:04:06.706454 (+1000197us) tablet_service.cc:2945] Iterator created
0114 03:04:06.706471 (+ 17us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823845706225 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823845208315 usec, L: 0 Physical time difference: 0.498s
0114 03:04:06.706876 (+ 405us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250114 03:04:06.771728 4159 tablet_service.cc:2970] error setting up scanner e6bdf248b5924f2ba69e5e5b42217ec5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823845771493 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823845208315 usec, L: 0 Physical time difference: 0.563s: new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:06.772112 4159 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:52220 (request call id 15) took 1001 ms. Trace:
I20250114 03:04:06.772178 4159 rpcz_store.cc:276] 0114 03:04:05.771092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:05.771139 (+ 47us) service_pool.cc:225] Handling call
0114 03:04:05.771300 (+ 161us) tablet_service.cc:2780] Created scanner e6bdf248b5924f2ba69e5e5b42217ec5 for tablet d5bcdea4a70943628babc9dfd8c94f1e, query id is f29e5d225b314cab9957bc0e7a2d8e4e
0114 03:04:05.771505 (+ 205us) tablet_service.cc:2920] Creating iterator
0114 03:04:05.771526 (+ 21us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:04:06.771701 (+1000175us) tablet_service.cc:2945] Iterator created
0114 03:04:06.771716 (+ 15us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823845771493 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823845208315 usec, L: 0 Physical time difference: 0.563s
0114 03:04:06.772097 (+ 381us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 03:04:06.951304 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:06.951457 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:06.952119 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:06.952433 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:06.952404 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:06.952948 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:06.953161 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:06.967160 4326 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=7, attempt_no=0}) took 1330 ms. Trace:
I20250114 03:04:06.967259 4326 rpcz_store.cc:276] 0114 03:04:05.636750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:05.636817 (+ 67us) service_pool.cc:225] Handling call
0114 03:04:06.967140 (+1330323us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:05.653828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:05.653962 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:05.653967 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:05.653968 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:05.654053 (+ 85us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:05.654063 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:05.654067 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:05.654116 (+ 49us) tablet.cc:698] Row locks acquired
0114 03:04:05.654117 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:05.654161 (+ 44us) write_op.cc:270] Start()
0114 03:04:06.548310 (+894149us) write_op.cc:276] Timestamp: P: 1736823845654090 usec, L: 0
0114 03:04:06.548315 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:06.548502 (+ 187us) log.cc:844] Serialized 354 byte log entry
0114 03:04:06.966560 (+418058us) op_driver.cc:464] REPLICATION: finished
0114 03:04:06.966800 (+ 240us) write_op.cc:301] APPLY: starting
0114 03:04:06.966816 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:06.966844 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:06.966857 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:06.966905 (+ 48us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:06.966927 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:06.966932 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:06.966989 (+ 57us) log.cc:844] Serialized 22 byte log entry
0114 03:04:06.967044 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:06.967059 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:06.967064 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":184,"num_ops":1,"prepare.queue_time_us":16609,"prepare.run_cpu_time_us":657,"prepare.run_wall_time_us":894729,"replication_time_us":418209,"thread_start_us":82,"threads_started":1}]]}
I20250114 03:04:07.629624 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:07.629781 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:07.630297 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:07.630653 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:07.630789 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:07.631177 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:07.631382 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:07.651279 4330 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=8, attempt_no=0}) took 1922 ms. Trace:
I20250114 03:04:07.651379 4330 rpcz_store.cc:276] 0114 03:04:05.728706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:05.728756 (+ 50us) service_pool.cc:225] Handling call
0114 03:04:07.651239 (+1922483us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:06.548587 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:06.548730 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:06.548738 (+ 8us) write_op.cc:435] Acquired schema lock
0114 03:04:06.548739 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:06.548858 (+ 119us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:06.548868 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:06.548870 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:06.548915 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:04:06.548916 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:06.548959 (+ 43us) write_op.cc:270] Start()
0114 03:04:07.182083 (+633124us) write_op.cc:276] Timestamp: P: 1736823846548896 usec, L: 0
0114 03:04:07.182086 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:07.182232 (+ 146us) log.cc:844] Serialized 354 byte log entry
0114 03:04:07.650624 (+468392us) op_driver.cc:464] REPLICATION: finished
0114 03:04:07.650864 (+ 240us) write_op.cc:301] APPLY: starting
0114 03:04:07.650900 (+ 36us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:07.650929 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:07.650944 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:07.650993 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:07.651017 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:07.651020 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:07.651081 (+ 61us) log.cc:844] Serialized 22 byte log entry
0114 03:04:07.651145 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:07.651159 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:07.651164 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":183,"num_ops":1,"prepare.queue_time_us":819456,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":633721,"raft.queue_time_us":134,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":3,"replication_time_us":468502,"thread_start_us":77,"threads_started":1}]]}
I20250114 03:04:07.682250 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:07.682382 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:07.682646 4067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:59569 (request call id 467) took 1448 ms. Trace:
I20250114 03:04:07.682898 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:07.682713 4067 rpcz_store.cc:276] 0114 03:04:06.234009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:06.234062 (+ 53us) service_pool.cc:225] Handling call
0114 03:04:06.234184 (+ 122us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:06.234245 (+ 61us) raft_consensus.cc:1529] Early marking committed up to index 6
0114 03:04:06.234464 (+ 219us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:06.234608 (+ 144us) write_op.cc:270] Start()
0114 03:04:07.170817 (+936209us) write_op.cc:276] Timestamp: P: 1736823845017420 usec, L: 0
0114 03:04:07.171249 (+ 432us) write_op.cc:270] Start()
0114 03:04:07.681422 (+510173us) write_op.cc:276] Timestamp: P: 1736823845208315 usec, L: 0
0114 03:04:07.681885 (+ 463us) log.cc:844] Serialized 708 byte log entry
0114 03:04:07.682007 (+ 122us) raft_consensus.cc:1640] Marking committed up to 8
0114 03:04:07.682199 (+ 192us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:07.682227 (+ 28us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:07.682576 (+ 349us) raft_consensus.cc:1680] finished
0114 03:04:07.682578 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:07.682626 (+ 48us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:07.171207 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:07.171307 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:07.171312 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:07.171313 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:07.171375 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:07.171386 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:07.171389 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:07.171433 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:04:07.171435 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:07.682049 (+510614us) op_driver.cc:464] REPLICATION: finished
0114 03:04:07.682350 (+ 301us) write_op.cc:301] APPLY: starting
0114 03:04:07.682367 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:07.682399 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:07.682400 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:07.682461 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:07.682488 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:07.682506 (+ 18us) write_op.cc:312] APPLY: finished
0114 03:04:07.682559 (+ 53us) log.cc:844] Serialized 22 byte log entry
0114 03:04:07.682596 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:07.682607 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:07.682612 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:07.682698 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:07.682701 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:07.682190 (+ 0us) op_driver.cc:464] REPLICATION: finished
0114 03:04:07.682317 (+ 127us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:07.682429 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:07.682434 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:07.682435 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:07.682505 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:07.682515 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:07.682517 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:07.682556 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:04:07.682557 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":309,"child_traces":[["op",{"apply.queue_time_us":246,"apply.run_cpu_time_us":343,"apply.run_wall_time_us":343,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":223,"prepare.run_cpu_time_us":314,"prepare.run_wall_time_us":313,"replication_time_us":1447420,"thread_start_us":169,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":771,"replication_time_us":510925,"thread_start_us":82,"threads_started":1}]]}
I20250114 03:04:07.683275 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:07.683631 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:07.684361 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:07.684517 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:08.438184 4159 tablet_service.cc:2970] error setting up scanner 804ac1cf4cde42a9b60c6632029a310d with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.412s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
W20250114 03:04:08.628921 4159 tablet_service.cc:2970] error setting up scanner 567428f761e948cc90cb77b1b6686583 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.603s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:08.653048 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:08.653194 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:08.653719 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:08.654119 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:08.654183 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:08.654603 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:08.654844 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:08.675963 4341 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=9, attempt_no=0}) took 1707 ms. Trace:
I20250114 03:04:08.676059 4341 rpcz_store.cc:276] 0114 03:04:06.968613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:06.968669 (+ 56us) service_pool.cc:225] Handling call
0114 03:04:08.675927 (+1707258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:07.182342 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:07.182470 (+ 128us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:07.182477 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:04:07.182480 (+ 3us) tablet.cc:659] Decoding operations
0114 03:04:07.182567 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:07.182580 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:07.182585 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:07.182631 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:04:07.182632 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:07.182674 (+ 42us) write_op.cc:270] Start()
0114 03:04:08.025839 (+843165us) write_op.cc:276] Timestamp: P: 1736823847182605 usec, L: 0
0114 03:04:08.025842 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:08.026015 (+ 173us) log.cc:844] Serialized 354 byte log entry
0114 03:04:08.675280 (+649265us) op_driver.cc:464] REPLICATION: finished
0114 03:04:08.675552 (+ 272us) write_op.cc:301] APPLY: starting
0114 03:04:08.675568 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:08.675595 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:08.675607 (+ 12us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:08.675656 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:08.675678 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:08.675682 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:08.675759 (+ 77us) log.cc:844] Serialized 22 byte log entry
0114 03:04:08.675819 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:08.675836 (+ 17us) write_op.cc:454] Released schema lock
0114 03:04:08.675841 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":191,"num_ops":1,"prepare.queue_time_us":213331,"prepare.run_cpu_time_us":662,"prepare.run_wall_time_us":843732,"replication_time_us":649398,"thread_start_us":99,"threads_started":1}]]}
I20250114 03:04:08.891340 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:08.891465 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:08.891896 4067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:59569 (request call id 468) took 1207 ms. Trace:
I20250114 03:04:08.892004 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:08.892400 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:08.892400 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:08.891971 4067 rpcz_store.cc:276] 0114 03:04:07.684286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:07.684471 (+ 185us) service_pool.cc:225] Handling call
0114 03:04:07.684598 (+ 127us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:07.684672 (+ 74us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 03:04:07.684674 (+ 2us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:07.684752 (+ 78us) write_op.cc:270] Start()
0114 03:04:08.275901 (+591149us) write_op.cc:276] Timestamp: P: 1736823845654090 usec, L: 0
0114 03:04:08.276298 (+ 397us) write_op.cc:270] Start()
0114 03:04:08.890467 (+614169us) write_op.cc:276] Timestamp: P: 1736823846548896 usec, L: 0
0114 03:04:08.890904 (+ 437us) log.cc:844] Serialized 708 byte log entry
0114 03:04:08.891044 (+ 140us) raft_consensus.cc:1640] Marking committed up to 10
0114 03:04:08.891281 (+ 237us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:08.891306 (+ 25us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:08.891848 (+ 542us) raft_consensus.cc:1680] finished
0114 03:04:08.891850 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:08.891881 (+ 31us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:08.276278 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:08.276398 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:08.276402 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:08.276404 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:08.276475 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:08.276485 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:08.276487 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:08.276524 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:04:08.276525 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:08.891083 (+614558us) op_driver.cc:464] REPLICATION: finished
0114 03:04:08.891303 (+ 220us) write_op.cc:301] APPLY: starting
0114 03:04:08.891321 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:08.891353 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:08.891355 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:08.891421 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:08.891521 (+ 100us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:08.891534 (+ 13us) write_op.cc:312] APPLY: finished
0114 03:04:08.891588 (+ 54us) log.cc:844] Serialized 22 byte log entry
0114 03:04:08.891714 (+ 126us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:08.891721 (+ 7us) write_op.cc:454] Released schema lock
0114 03:04:08.891724 (+ 3us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:08.891773 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:08.891775 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:08.890845 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:08.890945 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:08.890950 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:08.890951 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:08.891012 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:08.891021 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:08.891023 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:08.891058 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:08.891059 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:08.891200 (+ 141us) op_driver.cc:464] REPLICATION: finished
0114 03:04:08.891420 (+ 220us) write_op.cc:301] APPLY: starting
0114 03:04:08.891430 (+ 10us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:08.891446 (+ 16us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:08.891448 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:08.891477 (+ 29us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:08.891504 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:08.891527 (+ 23us) write_op.cc:312] APPLY: finished
0114 03:04:08.891588 (+ 61us) log.cc:844] Serialized 22 byte log entry
0114 03:04:08.891656 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:08.891667 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:08.891672 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:08.891752 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:08.891756 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":82,"threads_started":1,"wal-append.queue_time_us":149,"child_traces":[["op",{"apply.queue_time_us":155,"apply.run_cpu_time_us":421,"apply.run_wall_time_us":476,"mutex_wait_us":21,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":236,"prepare.run_cpu_time_us":320,"prepare.run_wall_time_us":320,"replication_time_us":1206312,"thread_start_us":138,"threads_started":2}],["op",{"apply.queue_time_us":182,"apply.run_cpu_time_us":310,"apply.run_wall_time_us":322,"mutex_wait_us":23,"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":256,"prepare.run_cpu_time_us":280,"prepare.run_wall_time_us":281,"replication_time_us":614891,"thread_start_us":137,"threads_started":2}]]}
I20250114 03:04:08.892863 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:08.893054 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:08.901271 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:08.901427 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:08.901897 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:08.902261 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:08.902261 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:08.902673 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:08.902876 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:08.921885 4341 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=10, attempt_no=0}) took 1269 ms. Trace:
I20250114 03:04:08.922001 4341 rpcz_store.cc:276] 0114 03:04:07.652647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:07.652704 (+ 57us) service_pool.cc:225] Handling call
0114 03:04:08.921865 (+1269161us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:08.026115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:08.026219 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:08.026227 (+ 8us) write_op.cc:435] Acquired schema lock
0114 03:04:08.026228 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:08.026299 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:08.026308 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:08.026311 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:08.026354 (+ 43us) tablet.cc:698] Row locks acquired
0114 03:04:08.026355 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:08.026391 (+ 36us) write_op.cc:270] Start()
0114 03:04:08.880534 (+854143us) write_op.cc:276] Timestamp: P: 1736823848026337 usec, L: 0
0114 03:04:08.880537 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:08.880728 (+ 191us) log.cc:844] Serialized 354 byte log entry
0114 03:04:08.921391 (+ 40663us) op_driver.cc:464] REPLICATION: finished
0114 03:04:08.921479 (+ 88us) write_op.cc:301] APPLY: starting
0114 03:04:08.921500 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:08.921535 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:08.921550 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:08.921608 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:08.921634 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:08.921638 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:08.921707 (+ 69us) log.cc:844] Serialized 22 byte log entry
0114 03:04:08.921770 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:08.921787 (+ 17us) write_op.cc:454] Released schema lock
0114 03:04:08.921793 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":38,"num_ops":1,"prepare.queue_time_us":373053,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":854679,"replication_time_us":40819}]]}
I20250114 03:04:09.673897 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:09.674032 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:09.674489 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:09.674821 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:09.674930 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 13 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:09.675383 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:09.689184 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:09.689337 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:09.689837 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:09.690200 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:09.690235 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:09.690601 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:09.711095 4352 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=11, attempt_no=0}) took 1033 ms. Trace:
I20250114 03:04:09.711179 4352 rpcz_store.cc:276] 0114 03:04:08.677425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:08.677487 (+ 62us) service_pool.cc:225] Handling call
0114 03:04:09.711062 (+1033575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:08.880869 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:08.881003 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:08.881008 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:08.881010 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:08.881079 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:08.881087 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:08.881089 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:08.881124 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:08.881125 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:08.881161 (+ 36us) write_op.cc:270] Start()
0114 03:04:09.158319 (+277158us) write_op.cc:276] Timestamp: P: 1736823848881129 usec, L: 0
0114 03:04:09.158324 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:09.158499 (+ 175us) log.cc:844] Serialized 354 byte log entry
0114 03:04:09.709320 (+550821us) op_driver.cc:464] REPLICATION: finished
0114 03:04:09.710652 (+ 1332us) write_op.cc:301] APPLY: starting
0114 03:04:09.710673 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:09.710713 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:09.710734 (+ 21us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:09.710802 (+ 68us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:09.710830 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:09.710838 (+ 8us) write_op.cc:312] APPLY: finished
0114 03:04:09.710917 (+ 79us) log.cc:844] Serialized 22 byte log entry
0114 03:04:09.710969 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:09.710982 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:09.710986 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1240,"num_ops":1,"prepare.queue_time_us":202982,"prepare.run_cpu_time_us":645,"prepare.run_wall_time_us":277725,"replication_time_us":550932,"thread_start_us":116,"threads_started":1}]]}
I20250114 03:04:10.236547 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:10.249341 4357 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=12, attempt_no=0}) took 1326 ms. Trace:
I20250114 03:04:10.249433 4357 rpcz_store.cc:276] 0114 03:04:08.923252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:08.923320 (+ 68us) service_pool.cc:225] Handling call
0114 03:04:10.249320 (+1326000us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:09.158593 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:09.158764 (+ 171us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:09.158770 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:09.158771 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:09.158851 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:09.158859 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:09.158863 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:09.158911 (+ 48us) tablet.cc:698] Row locks acquired
0114 03:04:09.158913 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:09.158956 (+ 43us) write_op.cc:270] Start()
0114 03:04:09.665893 (+506937us) write_op.cc:276] Timestamp: P: 1736823849158892 usec, L: 0
0114 03:04:09.665897 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:09.666114 (+ 217us) log.cc:844] Serialized 354 byte log entry
0114 03:04:10.248742 (+582628us) op_driver.cc:464] REPLICATION: finished
0114 03:04:10.249001 (+ 259us) write_op.cc:301] APPLY: starting
0114 03:04:10.249017 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:10.249043 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:10.249058 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:10.249109 (+ 51us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:10.249130 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:10.249133 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:10.249193 (+ 60us) log.cc:844] Serialized 22 byte log entry
0114 03:04:10.249240 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:10.249251 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:10.249255 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":208,"num_ops":1,"prepare.queue_time_us":234831,"prepare.run_cpu_time_us":1155,"prepare.run_wall_time_us":507582,"replication_time_us":582786,"thread_start_us":117,"threads_started":1}]]}
I20250114 03:04:10.540967 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:11.006245 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:11.006435 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:11.006982 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:11.007320 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:11.007534 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:11.007995 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:11.029191 4363 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=13, attempt_no=0}) took 1316 ms. Trace:
I20250114 03:04:11.029279 4363 rpcz_store.cc:276] 0114 03:04:09.712479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:09.712543 (+ 64us) service_pool.cc:225] Handling call
0114 03:04:11.029170 (+1316627us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:09.712972 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:09.713060 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:09.713063 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:09.713064 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:09.713120 (+ 56us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:09.713125 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:09.713128 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:09.713158 (+ 30us) tablet.cc:698] Row locks acquired
0114 03:04:09.713159 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:09.713184 (+ 25us) write_op.cc:270] Start()
0114 03:04:09.760318 (+ 47134us) write_op.cc:276] Timestamp: P: 1736823849713063 usec, L: 0
0114 03:04:09.760322 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:09.760515 (+ 193us) log.cc:844] Serialized 354 byte log entry
0114 03:04:11.028500 (+1267985us) op_driver.cc:464] REPLICATION: finished
0114 03:04:11.028775 (+ 275us) write_op.cc:301] APPLY: starting
0114 03:04:11.028796 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:11.028843 (+ 47us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:11.028860 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:11.028920 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:11.028944 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:11.028953 (+ 9us) write_op.cc:312] APPLY: finished
0114 03:04:11.029023 (+ 70us) log.cc:844] Serialized 22 byte log entry
0114 03:04:11.029079 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:11.029092 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:11.029096 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":214,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":531,"prepare.run_wall_time_us":47595,"replication_time_us":1268129,"thread_start_us":92,"threads_started":1}]]}
I20250114 03:04:11.079221 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:11.079386 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:11.079917 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:11.080322 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:11.080425 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 15 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:11.080776 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:11.232749 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:11.273416 4363 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=14, attempt_no=0}) took 1022 ms. Trace:
I20250114 03:04:11.273527 4363 rpcz_store.cc:276] 0114 03:04:10.250663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:10.250717 (+ 54us) service_pool.cc:225] Handling call
0114 03:04:11.273397 (+1022680us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:10.251105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:10.251191 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:10.251194 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:10.251195 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:10.251249 (+ 54us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:10.251254 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:10.251256 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:10.251283 (+ 27us) tablet.cc:698] Row locks acquired
0114 03:04:10.251284 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:10.251310 (+ 26us) write_op.cc:270] Start()
0114 03:04:10.556437 (+305127us) write_op.cc:276] Timestamp: P: 1736823850251242 usec, L: 0
0114 03:04:10.556441 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:10.556641 (+ 200us) log.cc:844] Serialized 354 byte log entry
0114 03:04:11.272752 (+716111us) op_driver.cc:464] REPLICATION: finished
0114 03:04:11.272964 (+ 212us) write_op.cc:301] APPLY: starting
0114 03:04:11.272982 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:11.273012 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:11.273027 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:11.273149 (+ 122us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:11.273175 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:11.273179 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:11.273243 (+ 64us) log.cc:844] Serialized 22 byte log entry
0114 03:04:11.273295 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:11.273312 (+ 17us) write_op.cc:454] Released schema lock
0114 03:04:11.273319 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":163,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":526,"prepare.run_wall_time_us":305586,"replication_time_us":716270}]]}
I20250114 03:04:11.652364 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:11.652534 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:11.653087 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:11.653482 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:11.653602 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 17 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:11.653982 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:11.713523 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:11.776387 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:12.074067 4321 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:12.074213 4321 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:12.074738 4321 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:12.075371 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:12.075369 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 18 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:12.075845 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:12.146762 4321 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:12.199249 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:12.199414 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:12.199891 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:12.200238 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:12.200284 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 19 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:12.200685 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:12.200892 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:12.792263 4159 tablet_service.cc:2970] error setting up scanner dc2e95600cfa48b5a8045a903eac5cba with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.644s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:12.884303 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:12.884474 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:12.885726 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:12.885879 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:12.886278 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:12.886417 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:12.886548 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:12.892956 4372 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=18, attempt_no=0}) took 1065 ms. Trace:
I20250114 03:04:12.893271 4372 rpcz_store.cc:276] 0114 03:04:11.827738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:11.827789 (+ 51us) service_pool.cc:225] Handling call
0114 03:04:12.892915 (+1065126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:11.993806 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:11.993921 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:11.993926 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:11.993928 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:11.994007 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:11.994016 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:11.994019 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:11.994060 (+ 41us) tablet.cc:698] Row locks acquired
0114 03:04:11.994061 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:11.994105 (+ 44us) write_op.cc:270] Start()
0114 03:04:12.664241 (+670136us) write_op.cc:276] Timestamp: P: 1736823851994058 usec, L: 0
0114 03:04:12.664244 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:12.664421 (+ 177us) log.cc:844] Serialized 354 byte log entry
0114 03:04:12.892378 (+227957us) op_driver.cc:464] REPLICATION: finished
0114 03:04:12.892559 (+ 181us) write_op.cc:301] APPLY: starting
0114 03:04:12.892572 (+ 13us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:12.892600 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:12.892615 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:12.892668 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:12.892691 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:12.892694 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:12.892754 (+ 60us) log.cc:844] Serialized 22 byte log entry
0114 03:04:12.892799 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:12.892809 (+ 10us) write_op.cc:454] Released schema lock
0114 03:04:12.892813 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":134,"num_ops":1,"prepare.queue_time_us":165683,"prepare.run_cpu_time_us":626,"prepare.run_wall_time_us":670680,"replication_time_us":228100,"thread_start_us":66,"threads_started":1}]]}
W20250114 03:04:13.238651 4159 tablet_service.cc:2970] error setting up scanner c7f731e118bb4246a5a1647ffdcaaee7 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.353s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:13.612239 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:13.612394 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:13.612916 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:13.613255 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:13.613494 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 20 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:13.613938 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:13.614111 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:13.673715 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:13.673871 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:13.674436 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:13.674875 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:13.674873 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 21 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:13.675297 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:13.675484 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:13.694132 4379 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=19, attempt_no=0}) took 1526 ms. Trace:
I20250114 03:04:13.694227 4379 rpcz_store.cc:276] 0114 03:04:12.167264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:12.167314 (+ 50us) service_pool.cc:225] Handling call
0114 03:04:13.694105 (+1526791us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:12.664521 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:12.664641 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:12.664647 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:12.664648 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:12.664745 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:12.664753 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:12.664755 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:12.664790 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:12.664791 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:12.664850 (+ 59us) write_op.cc:270] Start()
0114 03:04:13.327022 (+662172us) write_op.cc:276] Timestamp: P: 1736823852664801 usec, L: 0
0114 03:04:13.327025 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:13.327185 (+ 160us) log.cc:844] Serialized 354 byte log entry
0114 03:04:13.693179 (+365994us) op_driver.cc:464] REPLICATION: finished
0114 03:04:13.693565 (+ 386us) write_op.cc:301] APPLY: starting
0114 03:04:13.693586 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:13.693645 (+ 59us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:13.693709 (+ 64us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:13.693780 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:13.693811 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:13.693814 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:13.693907 (+ 93us) log.cc:844] Serialized 22 byte log entry
0114 03:04:13.693988 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:13.694001 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:13.694007 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":311,"num_ops":1,"prepare.queue_time_us":496815,"prepare.run_cpu_time_us":652,"prepare.run_wall_time_us":662745,"raft.queue_time_us":43,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":142,"replication_time_us":366102,"thread_start_us":143,"threads_started":1}]]}
I20250114 03:04:14.177635 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:14.177827 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:14.178339 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:14.178731 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:14.178746 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:14.179128 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:14.215513 4383 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=20, attempt_no=0}) took 1320 ms. Trace:
I20250114 03:04:14.215610 4383 rpcz_store.cc:276] 0114 03:04:12.894621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:12.894686 (+ 65us) service_pool.cc:225] Handling call
0114 03:04:14.215489 (+1320803us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:13.327297 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:13.327437 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:13.327442 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:13.327443 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:13.327519 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:13.327530 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:13.327533 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:13.327571 (+ 38us) tablet.cc:698] Row locks acquired
0114 03:04:13.327572 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:13.327606 (+ 34us) write_op.cc:270] Start()
0114 03:04:13.576778 (+249172us) write_op.cc:276] Timestamp: P: 1736823853327538 usec, L: 0
0114 03:04:13.576783 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:13.577051 (+ 268us) log.cc:844] Serialized 354 byte log entry
0114 03:04:14.214813 (+637762us) op_driver.cc:464] REPLICATION: finished
0114 03:04:14.215094 (+ 281us) write_op.cc:301] APPLY: starting
0114 03:04:14.215114 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:14.215142 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:14.215159 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:14.215218 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:14.215241 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:14.215244 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:14.215312 (+ 68us) log.cc:844] Serialized 22 byte log entry
0114 03:04:14.215376 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:14.215391 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:14.215396 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":213,"num_ops":1,"prepare.queue_time_us":432257,"prepare.run_cpu_time_us":744,"prepare.run_wall_time_us":249819,"replication_time_us":637982,"thread_start_us":94,"threads_started":1}]]}
I20250114 03:04:14.693403 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:14.709049 4383 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=21, attempt_no=0}) took 1013 ms. Trace:
I20250114 03:04:14.709152 4383 rpcz_store.cc:276] 0114 03:04:13.695523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:13.695604 (+ 81us) service_pool.cc:225] Handling call
0114 03:04:14.709031 (+1013427us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:13.696069 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:13.696162 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:13.696165 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:13.696166 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:13.696224 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:13.696231 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:13.696233 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:13.696266 (+ 33us) tablet.cc:698] Row locks acquired
0114 03:04:13.696267 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:13.696296 (+ 29us) write_op.cc:270] Start()
0114 03:04:13.819485 (+123189us) write_op.cc:276] Timestamp: P: 1736823853696259 usec, L: 0
0114 03:04:13.819489 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:13.819702 (+ 213us) log.cc:844] Serialized 354 byte log entry
0114 03:04:14.708467 (+888765us) op_driver.cc:464] REPLICATION: finished
0114 03:04:14.708558 (+ 91us) write_op.cc:301] APPLY: starting
0114 03:04:14.708577 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:14.708637 (+ 60us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:14.708654 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:14.708736 (+ 82us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:14.708767 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:14.708772 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:14.708867 (+ 95us) log.cc:844] Serialized 22 byte log entry
0114 03:04:14.708931 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:14.708944 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:14.708950 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":123717,"replication_time_us":888943}]]}
I20250114 03:04:14.901093 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:14.901242 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:14.901479 4067 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:59569 (request call id 1278) took 1287 ms. Trace:
I20250114 03:04:14.901546 4067 rpcz_store.cc:276] 0114 03:04:13.614198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:13.614239 (+ 41us) service_pool.cc:225] Handling call
0114 03:04:13.614372 (+ 133us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:13.614441 (+ 69us) raft_consensus.cc:1529] Early marking committed up to index 20
0114 03:04:13.614577 (+ 136us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:13.614672 (+ 95us) write_op.cc:270] Start()
0114 03:04:14.302858 (+688186us) write_op.cc:276] Timestamp: P: 1736823852664801 usec, L: 0
0114 03:04:14.303290 (+ 432us) write_op.cc:270] Start()
0114 03:04:14.900458 (+597168us) write_op.cc:276] Timestamp: P: 1736823853327538 usec, L: 0
0114 03:04:14.900901 (+ 443us) log.cc:844] Serialized 708 byte log entry
0114 03:04:14.901008 (+ 107us) raft_consensus.cc:1640] Marking committed up to 20
0114 03:04:14.901016 (+ 8us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:14.901048 (+ 32us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:14.901417 (+ 369us) raft_consensus.cc:1680] finished
0114 03:04:14.901418 (+ 1us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:14.901464 (+ 46us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:14.303258 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:14.303378 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:14.303385 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:04:14.303387 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:14.303455 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:14.303466 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:14.303469 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:14.303508 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:04:14.303509 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:04:14.900780 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:14.900939 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:14.900944 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:14.900945 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:14.901017 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:14.901025 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:14.901028 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:14.901070 (+ 42us) tablet.cc:698] Row locks acquired
0114 03:04:14.901071 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":52,"threads_started":1,"wal-append.queue_time_us":147,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":236,"prepare.run_cpu_time_us":325,"prepare.run_wall_time_us":325,"thread_start_us":95,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":195,"prepare.run_cpu_time_us":376,"prepare.run_wall_time_us":375,"thread_start_us":78,"threads_started":1}]]}
I20250114 03:04:14.902156 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:14.902307 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 22 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:14.901763 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:14.902683 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:14.902839 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:15.006415 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:15.006556 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:15.007143 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:15.007547 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:15.007618 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:15.008121 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:15.008281 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:15.387816 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:15.387959 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:15.388504 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:15.389066 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:15.389061 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 24 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:15.389590 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:15.389781 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:15.464165 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:15.464326 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:15.464874 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:15.465278 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:15.465327 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:15.465821 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:15.465983 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:15.688169 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:15.688302 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:15.688741 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:15.689096 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:15.689258 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 25 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:15.689654 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:15.690341 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:15.947410 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:15.947561 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:15.948078 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:15.948483 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:15.948483 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:15.949173 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:15.949380 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:16.016361 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:16.016523 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:16.017009 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:16.017271 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:16.017403 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 26 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:16.017840 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:16.018033 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:16.681001 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:16.681138 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:16.681625 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:16.682008 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:16.682085 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:16.682555 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:16.682714 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:16.731699 4397 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=25, attempt_no=0}) took 1225 ms. Trace:
I20250114 03:04:16.731817 4397 rpcz_store.cc:276] 0114 03:04:15.505913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:15.505981 (+ 68us) service_pool.cc:225] Handling call
0114 03:04:16.731679 (+1225698us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:15.718667 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:15.718784 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:15.718789 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:15.718790 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:15.718873 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:15.718881 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:15.718884 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:15.718926 (+ 42us) tablet.cc:698] Row locks acquired
0114 03:04:15.718927 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:15.718975 (+ 48us) write_op.cc:270] Start()
0114 03:04:16.227097 (+508122us) write_op.cc:276] Timestamp: P: 1736823855718922 usec, L: 0
0114 03:04:16.227102 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:16.227271 (+ 169us) log.cc:844] Serialized 354 byte log entry
0114 03:04:16.731071 (+503800us) op_driver.cc:464] REPLICATION: finished
0114 03:04:16.731290 (+ 219us) write_op.cc:301] APPLY: starting
0114 03:04:16.731307 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:16.731345 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:16.731361 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:16.731426 (+ 65us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:16.731454 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:16.731458 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:16.731518 (+ 60us) log.cc:844] Serialized 22 byte log entry
0114 03:04:16.731577 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:16.731591 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:16.731597 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":171,"mutex_wait_us":41,"num_ops":1,"prepare.queue_time_us":212357,"prepare.run_cpu_time_us":891,"prepare.run_wall_time_us":508962,"raft.queue_time_us":207,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":193,"replication_time_us":503930,"spinlock_wait_cycles":538112,"thread_start_us":234,"threads_started":2}]]}
I20250114 03:04:16.834007 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:16.834141 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:16.834635 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:16.835042 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:16.835072 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 27 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:16.835836 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:16.836000 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:17.080631 4159 tablet_service.cc:2970] error setting up scanner d9c1048a3482433e80878b26ae770e83 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.246s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:17.402261 4159 tablet_service.cc:2970] error setting up scanner 62f1e11542dd4dec9e5e5fccfc675545 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.567s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:17.586233 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:17.586378 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:17.586833 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:17.587258 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:17.587409 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:17.587803 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:17.597057 4404 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=26, attempt_no=0}) took 1638 ms. Trace:
I20250114 03:04:17.597174 4404 rpcz_store.cc:276] 0114 03:04:15.959004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:15.959078 (+ 74us) service_pool.cc:225] Handling call
0114 03:04:17.597034 (+1637956us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:16.227649 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:16.227768 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:16.227774 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:16.227775 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:16.227857 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:16.227870 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:16.227874 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:16.227932 (+ 58us) tablet.cc:698] Row locks acquired
0114 03:04:16.227933 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:16.227976 (+ 43us) write_op.cc:270] Start()
0114 03:04:16.984123 (+756147us) write_op.cc:276] Timestamp: P: 1736823856227928 usec, L: 0
0114 03:04:16.984127 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:16.984319 (+ 192us) log.cc:844] Serialized 354 byte log entry
0114 03:04:17.596398 (+612079us) op_driver.cc:464] REPLICATION: finished
0114 03:04:17.596656 (+ 258us) write_op.cc:301] APPLY: starting
0114 03:04:17.596671 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:17.596698 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:17.596712 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:17.596768 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:17.596790 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:17.596793 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:17.596874 (+ 81us) log.cc:844] Serialized 22 byte log entry
0114 03:04:17.596940 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:17.596951 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:17.596955 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":201,"num_ops":1,"prepare.queue_time_us":268212,"prepare.run_cpu_time_us":676,"prepare.run_wall_time_us":756734,"replication_time_us":612230,"thread_start_us":83,"threads_started":1}]]}
I20250114 03:04:17.597653 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:17.900171 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:17.900328 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:17.900954 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:17.901399 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:17.901458 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 28 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:17.901906 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:18.111099 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:18.111265 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:18.112063 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:18.112272 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:18.112272 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 30 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:18.112718 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:18.112952 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:18.579478 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:19.353266 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:19.353430 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:19.354004 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:19.354465 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:19.354465 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:19.354983 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:19.374307 4418 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=29, attempt_no=0}) took 1737 ms. Trace:
I20250114 03:04:19.374419 4418 rpcz_store.cc:276] 0114 03:04:17.636406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:17.636470 (+ 64us) service_pool.cc:225] Handling call
0114 03:04:19.374281 (+1737811us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:17.770792 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:17.770906 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:17.770911 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:17.770912 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:17.770991 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:17.771002 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:17.771005 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:17.771049 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:04:17.771052 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:04:17.771095 (+ 43us) write_op.cc:270] Start()
0114 03:04:18.586219 (+815124us) write_op.cc:276] Timestamp: P: 1736823857771046 usec, L: 0
0114 03:04:18.586222 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:18.586420 (+ 198us) log.cc:844] Serialized 354 byte log entry
0114 03:04:19.373540 (+787120us) op_driver.cc:464] REPLICATION: finished
0114 03:04:19.373856 (+ 316us) write_op.cc:301] APPLY: starting
0114 03:04:19.373877 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:19.373906 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:19.373923 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:19.373994 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:19.374019 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:19.374022 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:19.374093 (+ 71us) log.cc:844] Serialized 22 byte log entry
0114 03:04:19.374187 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:19.374200 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:19.374204 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":247,"mutex_wait_us":21,"num_ops":1,"prepare.queue_time_us":133966,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":815818,"raft.queue_time_us":325,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":195,"replication_time_us":787258,"spinlock_wait_cycles":90624,"thread_start_us":211,"threads_started":2}]]}
I20250114 03:04:19.386385 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:19.386523 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:19.387048 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:19.387802 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:19.387934 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 31 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:19.388370 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:19.432897 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:19.444487 4418 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=30, attempt_no=0}) took 1281 ms. Trace:
I20250114 03:04:19.444586 4418 rpcz_store.cc:276] 0114 03:04:18.162727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:18.162791 (+ 64us) service_pool.cc:225] Handling call
0114 03:04:19.444468 (+1281677us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:18.586636 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:18.586752 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:18.586756 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:18.586757 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:18.586812 (+ 55us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:18.586818 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:18.586820 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:18.586854 (+ 34us) tablet.cc:698] Row locks acquired
0114 03:04:18.586855 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:18.586885 (+ 30us) write_op.cc:270] Start()
0114 03:04:19.113106 (+526221us) write_op.cc:276] Timestamp: P: 1736823858586861 usec, L: 0
0114 03:04:19.113110 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:19.113329 (+ 219us) log.cc:844] Serialized 354 byte log entry
0114 03:04:19.443938 (+330609us) op_driver.cc:464] REPLICATION: finished
0114 03:04:19.444065 (+ 127us) write_op.cc:301] APPLY: starting
0114 03:04:19.444086 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:19.444120 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:19.444143 (+ 23us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:19.444216 (+ 73us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:19.444243 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:19.444246 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:19.444320 (+ 74us) log.cc:844] Serialized 22 byte log entry
0114 03:04:19.444375 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:19.444387 (+ 12us) write_op.cc:454] Released schema lock
0114 03:04:19.444392 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":67,"num_ops":1,"prepare.queue_time_us":423495,"prepare.run_cpu_time_us":660,"prepare.run_wall_time_us":526757,"replication_time_us":330777}]]}
I20250114 03:04:19.877326 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:19.877447 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:19.877986 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:19.878360 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:19.878432 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:19.878945 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:19.879137 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:20.108063 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:20.233341 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:20.233487 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:20.233994 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:20.234275 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:20.234340 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 33 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:20.234735 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:20.234930 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:20.666786 4159 tablet_service.cc:2970] error setting up scanner c9d92196b61c47a29358cf33e61a92c5 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.788s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:20.733538 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:20.733692 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:20.734200 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:20.734687 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:20.734723 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:20.735219 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:20.735540 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:20.747588 4426 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=32, attempt_no=0}) took 1301 ms. Trace:
I20250114 03:04:20.747789 4426 rpcz_store.cc:276] 0114 03:04:19.446064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:19.446127 (+ 63us) service_pool.cc:225] Handling call
0114 03:04:20.747555 (+1301428us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:19.446565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:19.446665 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:19.446668 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:19.446669 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:19.446735 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:19.446742 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:19.446745 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:19.446784 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:04:19.446785 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:19.446816 (+ 31us) write_op.cc:270] Start()
0114 03:04:20.251955 (+805139us) write_op.cc:276] Timestamp: P: 1736823859446760 usec, L: 0
0114 03:04:20.251958 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:20.252126 (+ 168us) log.cc:844] Serialized 354 byte log entry
0114 03:04:20.254721 (+ 2595us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x55e0418517d0. stack: 00007f18baf87373 00007f18baf87965 00007f18baaf089d 000055e016f8ebee 000055e016f8ec41 00007f18bd8da909 00007f18bd8da877 00007f18bcabfc13 00007f18bcabefb9 00007f18bcac3c98 000055e01701bbbd 00007f18bafaac0b 00007f18bafab4f4 00007f18baface40 000055e01701bbbd 00007f18baf9a658
0114 03:04:20.746830 (+492109us) op_driver.cc:464] REPLICATION: finished
0114 03:04:20.747123 (+ 293us) write_op.cc:301] APPLY: starting
0114 03:04:20.747154 (+ 31us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:20.747188 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:20.747189 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:20.747244 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:20.747272 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:20.747284 (+ 12us) write_op.cc:312] APPLY: finished
0114 03:04:20.747364 (+ 80us) log.cc:844] Serialized 22 byte log entry
0114 03:04:20.747456 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:20.747472 (+ 16us) write_op.cc:454] Released schema lock
0114 03:04:20.747479 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":167,"num_ops":1,"prepare.queue_time_us":42,"prepare.run_cpu_time_us":586,"prepare.run_wall_time_us":805643,"raft.queue_time_us":266,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":2277,"replication_time_us":494835,"spinlock_wait_cycles":4366592,"thread_start_us":74,"threads_started":1}]]}
W20250114 03:04:20.988091 4159 tablet_service.cc:2970] error setting up scanner e3f89ce09429468fbc277b63634af78c with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.253s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:21.026000 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:21.026150 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:21.026607 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:21.026890 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:21.027005 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 34 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:21.027362 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:21.027510 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:21.503032 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:21.503182 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:21.503717 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:21.504212 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:21.504227 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:21.504603 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:21.504861 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:21.534715 4431 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=33, attempt_no=0}) took 1620 ms. Trace:
I20250114 03:04:21.534821 4431 rpcz_store.cc:276] 0114 03:04:19.914111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:19.914165 (+ 54us) service_pool.cc:225] Handling call
0114 03:04:21.534683 (+1620518us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:20.252247 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:20.252374 (+ 127us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:20.252379 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:20.252380 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:20.252453 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:20.252462 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:20.252465 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:20.252509 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:04:20.252512 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:04:20.252558 (+ 46us) write_op.cc:270] Start()
0114 03:04:21.223695 (+971137us) write_op.cc:276] Timestamp: P: 1736823860252497 usec, L: 0
0114 03:04:21.223698 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:21.223851 (+ 153us) log.cc:844] Serialized 354 byte log entry
0114 03:04:21.534064 (+310213us) op_driver.cc:464] REPLICATION: finished
0114 03:04:21.534266 (+ 202us) write_op.cc:301] APPLY: starting
0114 03:04:21.534285 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:21.534318 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:21.534331 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:21.534415 (+ 84us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:21.534439 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:21.534443 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:21.534517 (+ 74us) log.cc:844] Serialized 22 byte log entry
0114 03:04:21.534589 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:21.534602 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:21.534606 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":145,"num_ops":1,"prepare.queue_time_us":337755,"prepare.run_cpu_time_us":639,"prepare.run_wall_time_us":971693,"raft.queue_time_us":87,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":173,"replication_time_us":310326,"thread_start_us":71,"threads_started":1}]]}
I20250114 03:04:21.893810 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:21.893980 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:21.894538 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:21.895004 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:21.895030 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 35 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:21.895460 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:22.068107 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:22.086076 4435 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=34, attempt_no=0}) took 1337 ms. Trace:
I20250114 03:04:22.086175 4435 rpcz_store.cc:276] 0114 03:04:20.749028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:20.749082 (+ 54us) service_pool.cc:225] Handling call
0114 03:04:22.086056 (+1336974us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:21.223968 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:21.224097 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:21.224102 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:21.224104 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:21.224179 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:21.224188 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:21.224191 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:21.224237 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:04:21.224238 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:21.224274 (+ 36us) write_op.cc:270] Start()
0114 03:04:21.804406 (+580132us) write_op.cc:276] Timestamp: P: 1736823861224240 usec, L: 0
0114 03:04:21.804409 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:21.804598 (+ 189us) log.cc:844] Serialized 354 byte log entry
0114 03:04:22.085412 (+280814us) op_driver.cc:464] REPLICATION: finished
0114 03:04:22.085639 (+ 227us) write_op.cc:301] APPLY: starting
0114 03:04:22.085656 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:22.085694 (+ 38us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:22.085711 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:22.085776 (+ 65us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:22.085808 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:22.085812 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:22.085882 (+ 70us) log.cc:844] Serialized 22 byte log entry
0114 03:04:22.085941 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:22.085956 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:22.085975 (+ 19us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":170,"num_ops":1,"prepare.queue_time_us":474591,"prepare.run_cpu_time_us":635,"prepare.run_wall_time_us":580702,"replication_time_us":280945,"thread_start_us":87,"threads_started":1}]]}
I20250114 03:04:22.332716 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:22.332877 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:22.333361 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:22.333698 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:22.334069 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 36 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:22.334540 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:22.334789 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:22.609800 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:22.609951 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:22.610530 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:22.610980 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:22.610975 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:22.611441 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:22.619457 4441 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=35, attempt_no=0}) took 1083 ms. Trace:
I20250114 03:04:22.619539 4441 rpcz_store.cc:276] 0114 03:04:21.536082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:21.536143 (+ 61us) service_pool.cc:225] Handling call
0114 03:04:22.619438 (+1083295us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:21.804698 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:21.804813 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:21.804844 (+ 31us) write_op.cc:435] Acquired schema lock
0114 03:04:21.804847 (+ 3us) tablet.cc:659] Decoding operations
0114 03:04:21.804929 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:21.804940 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:21.804944 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:21.804994 (+ 50us) tablet.cc:698] Row locks acquired
0114 03:04:21.804995 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:21.805041 (+ 46us) write_op.cc:270] Start()
0114 03:04:22.432201 (+627160us) write_op.cc:276] Timestamp: P: 1736823861804996 usec, L: 0
0114 03:04:22.432206 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:22.432399 (+ 193us) log.cc:844] Serialized 354 byte log entry
0114 03:04:22.618865 (+186466us) op_driver.cc:464] REPLICATION: finished
0114 03:04:22.619105 (+ 240us) write_op.cc:301] APPLY: starting
0114 03:04:22.619122 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:22.619150 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:22.619165 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:22.619221 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:22.619244 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:22.619247 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:22.619308 (+ 61us) log.cc:844] Serialized 22 byte log entry
0114 03:04:22.619357 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:22.619368 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:22.619373 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":181,"num_ops":1,"prepare.queue_time_us":268180,"prepare.run_cpu_time_us":733,"prepare.run_wall_time_us":627787,"raft.queue_time_us":43,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":186615,"thread_start_us":81,"threads_started":1}]]}
I20250114 03:04:22.650470 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:22.882557 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:22.882755 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:22.883337 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:22.883841 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:22.883939 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 39 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:22.884413 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:22.884872 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:23.231572 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:23.231710 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:23.232311 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:23.232676 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:23.232758 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 37 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:23.233180 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:23.854132 4271 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:23.854302 4271 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:23.854820 4271 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:23.855165 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:23.855629 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 40 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:23.856066 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:23.885387 4448 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=38, attempt_no=0}) took 1204 ms. Trace:
I20250114 03:04:23.885509 4448 rpcz_store.cc:276] 0114 03:04:22.681310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:22.681367 (+ 57us) service_pool.cc:225] Handling call
0114 03:04:23.885365 (+1203998us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:22.681834 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:22.681931 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:22.681934 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:22.681935 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:22.682002 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:22.682009 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:22.682011 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:22.682051 (+ 40us) tablet.cc:698] Row locks acquired
0114 03:04:22.682054 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:04:22.682092 (+ 38us) write_op.cc:270] Start()
0114 03:04:23.234197 (+552105us) write_op.cc:276] Timestamp: P: 1736823862682055 usec, L: 0
0114 03:04:23.234200 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:23.234323 (+ 123us) log.cc:844] Serialized 354 byte log entry
0114 03:04:23.884628 (+650305us) op_driver.cc:464] REPLICATION: finished
0114 03:04:23.884980 (+ 352us) write_op.cc:301] APPLY: starting
0114 03:04:23.885001 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:23.885031 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:23.885049 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:23.885112 (+ 63us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:23.885137 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:23.885140 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:23.885214 (+ 74us) log.cc:844] Serialized 22 byte log entry
0114 03:04:23.885268 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:23.885282 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:23.885286 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":273,"num_ops":1,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":496,"prepare.run_wall_time_us":552549,"replication_time_us":650370,"thread_start_us":110,"threads_started":1}]]}
I20250114 03:04:23.932008 4271 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:23.977535 4448 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=39, attempt_no=0}) took 1045 ms. Trace:
I20250114 03:04:23.977643 4448 rpcz_store.cc:276] 0114 03:04:22.932083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:22.932144 (+ 61us) service_pool.cc:225] Handling call
0114 03:04:23.977515 (+1045371us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:23.234405 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:23.234517 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:23.234520 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:23.234522 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:23.234603 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:23.234611 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:23.234614 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:23.234655 (+ 41us) tablet.cc:698] Row locks acquired
0114 03:04:23.234656 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:23.234692 (+ 36us) write_op.cc:270] Start()
0114 03:04:23.577863 (+343171us) write_op.cc:276] Timestamp: P: 1736823863234653 usec, L: 0
0114 03:04:23.577867 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:23.578076 (+ 209us) log.cc:844] Serialized 354 byte log entry
0114 03:04:23.977012 (+398936us) op_driver.cc:464] REPLICATION: finished
0114 03:04:23.977125 (+ 113us) write_op.cc:301] APPLY: starting
0114 03:04:23.977153 (+ 28us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:23.977190 (+ 37us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:23.977203 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:23.977265 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:23.977290 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:23.977293 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:23.977355 (+ 62us) log.cc:844] Serialized 22 byte log entry
0114 03:04:23.977414 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:23.977429 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:23.977434 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":301907,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":343738,"replication_time_us":399093}]]}
W20250114 03:04:23.996786 4159 tablet_service.cc:2970] error setting up scanner de2fa4585a8749b4ae1b458673bac508 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.763s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:24.822309 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:24.822639 4198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:49933 (request call id 4659) took 1589 ms. Trace:
I20250114 03:04:24.822741 4198 rpcz_store.cc:276] 0114 03:04:23.232790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:23.233115 (+ 325us) service_pool.cc:225] Handling call
0114 03:04:23.233271 (+ 156us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:23.233354 (+ 83us) raft_consensus.cc:1529] Early marking committed up to index 37
0114 03:04:23.233535 (+ 181us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:23.233671 (+ 136us) write_op.cc:270] Start()
0114 03:04:24.180874 (+947203us) write_op.cc:276] Timestamp: P: 1736823862432857 usec, L: 0
0114 03:04:24.181251 (+ 377us) write_op.cc:270] Start()
0114 03:04:24.821418 (+640167us) write_op.cc:276] Timestamp: P: 1736823862621564 usec, L: 0
0114 03:04:24.821865 (+ 447us) log.cc:844] Serialized 708 byte log entry
0114 03:04:24.821977 (+ 112us) raft_consensus.cc:1640] Marking committed up to 39
0114 03:04:24.822227 (+ 250us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:24.822256 (+ 29us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:24.822484 (+ 228us) raft_consensus.cc:1680] finished
0114 03:04:24.822485 (+ 1us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:24.822536 (+ 51us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:24.181279 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:24.181390 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:24.181393 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:24.181395 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:24.181467 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:24.181479 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:24.181482 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:24.181529 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:04:24.181531 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:24.822035 (+640504us) op_driver.cc:464] REPLICATION: finished
0114 03:04:24.822232 (+ 197us) write_op.cc:301] APPLY: starting
0114 03:04:24.822250 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:24.822282 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:24.822285 (+ 3us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:24.822343 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
Related trace 'op':
0114 03:04:24.821814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:24.821928 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:24.821933 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:24.821934 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:24.822006 (+ 72us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:24.822016 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:24.822018 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:24.822056 (+ 38us) tablet.cc:698] Row locks acquired
0114 03:04:24.822057 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:24.822146 (+ 89us) op_driver.cc:464] REPLICATION: finished
0114 03:04:24.822292 (+ 146us) write_op.cc:301] APPLY: starting
0114 03:04:24.822299 (+ 7us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:24.822312 (+ 13us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:24.822313 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:24.822343 (+ 30us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:24.822375 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:24.822403 (+ 28us) write_op.cc:312] APPLY: finished
0114 03:04:24.822494 (+ 91us) log.cc:844] Serialized 22 byte log entry
0114 03:04:24.822540 (+ 46us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:24.822550 (+ 10us) write_op.cc:454] Released schema lock
0114 03:04:24.822555 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:24.822632 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:24.822634 (+ 2us) write_op.cc:454] Released schema lock
Metrics: {"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":140,"child_traces":[["op",{"apply.queue_time_us":155,"apply.run_cpu_time_us":432,"apply.run_wall_time_us":977,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":251,"prepare.run_cpu_time_us":337,"prepare.run_wall_time_us":337,"replication_time_us":1588343,"thread_start_us":150,"threads_started":2}],["op",{"apply.queue_time_us":115,"apply.run_cpu_time_us":328,"apply.run_wall_time_us":328,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":220,"prepare.run_cpu_time_us":319,"prepare.run_wall_time_us":320,"replication_time_us":640885,"thread_start_us":145,"threads_started":2}]]}
I20250114 03:04:25.002490 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:25.002650 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:25.003181 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:25.003587 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:25.003646 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:25.004047 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:25.004248 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:25.042584 4459 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=40, attempt_no=0}) took 1155 ms. Trace:
I20250114 03:04:25.042699 4459 rpcz_store.cc:276] 0114 03:04:23.886869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:23.886927 (+ 58us) service_pool.cc:225] Handling call
0114 03:04:25.042538 (+1155611us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:23.887378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:23.887475 (+ 97us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:23.887478 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:23.887479 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:23.887549 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:23.887555 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:23.887557 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:23.887591 (+ 34us) tablet.cc:698] Row locks acquired
0114 03:04:23.887592 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:23.887625 (+ 33us) write_op.cc:270] Start()
0114 03:04:24.766775 (+879150us) write_op.cc:276] Timestamp: P: 1736823863887580 usec, L: 0
0114 03:04:24.766779 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:24.766943 (+ 164us) log.cc:844] Serialized 354 byte log entry
0114 03:04:25.041612 (+274669us) op_driver.cc:464] REPLICATION: finished
0114 03:04:25.041908 (+ 296us) write_op.cc:301] APPLY: starting
0114 03:04:25.041927 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:25.041955 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:25.041998 (+ 43us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:25.042077 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:25.042128 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:25.042134 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:04:25.042248 (+ 114us) log.cc:844] Serialized 22 byte log entry
0114 03:04:25.042398 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:25.042411 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:25.042417 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":235,"mutex_wait_us":51,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":721,"prepare.run_wall_time_us":879811,"raft.queue_time_us":172,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":274794,"spinlock_wait_cycles":3840,"thread_start_us":182,"threads_started":2}]]}
W20250114 03:04:25.086010 4159 tablet_service.cc:2970] error setting up scanner 88ba7c269c9d46549b8c7f8d002ee2ac with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.264s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:25.955833 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:25.955964 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:25.956440 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:25.956614 4198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:49933 (request call id 4660) took 1132 ms. Trace:
I20250114 03:04:25.956998 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:25.957013 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 42 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:25.956694 4198 rpcz_store.cc:276] 0114 03:04:24.824528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:24.824588 (+ 60us) service_pool.cc:225] Handling call
0114 03:04:24.824765 (+ 177us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 03:04:24.824863 (+ 98us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 03:04:24.824865 (+ 2us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 03:04:24.824968 (+ 103us) write_op.cc:270] Start()
0114 03:04:25.257127 (+432159us) write_op.cc:276] Timestamp: P: 1736823862682055 usec, L: 0
0114 03:04:25.257374 (+ 247us) write_op.cc:270] Start()
0114 03:04:25.390522 (+133148us) write_op.cc:276] Timestamp: P: 1736823863234653 usec, L: 0
0114 03:04:25.390765 (+ 243us) write_op.cc:270] Start()
0114 03:04:25.954950 (+564185us) write_op.cc:276] Timestamp: P: 1736823863887580 usec, L: 0
0114 03:04:25.955388 (+ 438us) log.cc:844] Serialized 1062 byte log entry
0114 03:04:25.955491 (+ 103us) raft_consensus.cc:1640] Marking committed up to 41
0114 03:04:25.955767 (+ 276us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:25.955798 (+ 31us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:25.956545 (+ 747us) raft_consensus.cc:1680] finished
0114 03:04:25.956549 (+ 4us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:25.956601 (+ 52us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:25.257294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:25.257404 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:25.257408 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:25.257410 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:25.257478 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:25.257487 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:25.257490 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:25.257537 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:04:25.257539 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:25.955533 (+697994us) op_driver.cc:464] REPLICATION: finished
0114 03:04:25.955713 (+ 180us) write_op.cc:301] APPLY: starting
0114 03:04:25.955742 (+ 29us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:25.955775 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:25.955776 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:25.955837 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:25.955867 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:25.955891 (+ 24us) write_op.cc:312] APPLY: finished
0114 03:04:25.955956 (+ 65us) log.cc:844] Serialized 22 byte log entry
0114 03:04:25.956006 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:25.956017 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:25.956022 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:25.956101 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:25.956103 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:25.390706 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:25.390811 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:25.390816 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:25.390817 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:25.390881 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:25.390890 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:25.390892 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:25.390927 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:25.390928 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:25.955657 (+564729us) op_driver.cc:464] REPLICATION: finished
0114 03:04:25.955867 (+ 210us) write_op.cc:301] APPLY: starting
0114 03:04:25.955876 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:25.955892 (+ 16us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:25.955893 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:25.955928 (+ 35us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:25.955943 (+ 15us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:25.955953 (+ 10us) write_op.cc:312] APPLY: finished
0114 03:04:25.955996 (+ 43us) log.cc:844] Serialized 22 byte log entry
0114 03:04:25.956016 (+ 20us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:25.956020 (+ 4us) write_op.cc:454] Released schema lock
0114 03:04:25.956023 (+ 3us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:25.956093 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:25.956096 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:25.955321 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:25.955426 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:25.955431 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:25.955432 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:25.955497 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:25.955507 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:25.955509 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:25.955560 (+ 51us) tablet.cc:698] Row locks acquired
0114 03:04:25.955562 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":135,"child_traces":[["op",{"apply.queue_time_us":142,"apply.run_cpu_time_us":387,"apply.run_wall_time_us":386,"mutex_wait_us":16,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":321,"prepare.run_wall_time_us":320,"replication_time_us":1130544,"spinlock_wait_cycles":60288,"thread_start_us":63,"threads_started":1}],["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":219,"apply.run_wall_time_us":219,"num_ops":1,"partition_lock_wait_us":4,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":301,"prepare.run_wall_time_us":300,"replication_time_us":698272,"thread_start_us":49,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":218,"prepare.run_cpu_time_us":305,"prepare.run_wall_time_us":305,"thread_start_us":82,"threads_started":1}]]}
I20250114 03:04:25.957513 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:25.957684 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:26.160336 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:26.160482 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:26.161190 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:26.161631 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:26.161711 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 43 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:26.162206 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:26.206751 4469 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=41, attempt_no=0}) took 2227 ms. Trace:
I20250114 03:04:26.206864 4469 rpcz_store.cc:276] 0114 03:04:23.979060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:23.979114 (+ 54us) service_pool.cc:225] Handling call
0114 03:04:26.206717 (+2227603us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:24.767221 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:24.767334 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:24.767338 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:24.767339 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:24.767401 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:24.767409 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:24.767411 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:24.767446 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:24.767447 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:24.767482 (+ 35us) write_op.cc:270] Start()
0114 03:04:25.429615 (+662133us) write_op.cc:276] Timestamp: P: 1736823864767446 usec, L: 0
0114 03:04:25.429618 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:25.429846 (+ 228us) log.cc:844] Serialized 354 byte log entry
0114 03:04:26.206046 (+776200us) op_driver.cc:464] REPLICATION: finished
0114 03:04:26.206325 (+ 279us) write_op.cc:301] APPLY: starting
0114 03:04:26.206345 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:26.206374 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:26.206393 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:26.206457 (+ 64us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:26.206482 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:26.206485 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:26.206559 (+ 74us) log.cc:844] Serialized 22 byte log entry
0114 03:04:26.206613 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:26.206627 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:26.206631 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":205,"num_ops":1,"prepare.queue_time_us":787775,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":662733,"raft.queue_time_us":76,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":168,"replication_time_us":776379,"spinlock_wait_cycles":112000,"thread_start_us":93,"threads_started":1}]]}
I20250114 03:04:26.715102 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:26.728288 4474 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=42, attempt_no=0}) took 1684 ms. Trace:
I20250114 03:04:26.728392 4474 rpcz_store.cc:276] 0114 03:04:25.044084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:25.044148 (+ 64us) service_pool.cc:225] Handling call
0114 03:04:26.728260 (+1684112us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:25.429998 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:25.430135 (+ 137us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:25.430142 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:04:25.430143 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:25.430225 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:25.430233 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:25.430235 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:25.430271 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:04:25.430272 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:25.430311 (+ 39us) write_op.cc:270] Start()
0114 03:04:25.785472 (+355161us) write_op.cc:276] Timestamp: P: 1736823865430256 usec, L: 0
0114 03:04:25.785476 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:25.785768 (+ 292us) log.cc:844] Serialized 354 byte log entry
0114 03:04:26.727508 (+941740us) op_driver.cc:464] REPLICATION: finished
0114 03:04:26.727824 (+ 316us) write_op.cc:301] APPLY: starting
0114 03:04:26.727845 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:26.727874 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:26.727891 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:26.727971 (+ 80us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:26.727997 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:26.728000 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:26.728075 (+ 75us) log.cc:844] Serialized 22 byte log entry
0114 03:04:26.728139 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:26.728155 (+ 16us) write_op.cc:454] Released schema lock
0114 03:04:26.728161 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":254,"num_ops":1,"prepare.queue_time_us":385502,"prepare.run_cpu_time_us":790,"prepare.run_wall_time_us":355855,"replication_time_us":941971,"thread_start_us":171,"threads_started":1}]]}
I20250114 03:04:27.021976 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:27.022137 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:27.022464 4198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:49933 (request call id 4661) took 1064 ms. Trace:
I20250114 03:04:27.022681 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:27.022539 4198 rpcz_store.cc:276] 0114 03:04:25.958229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:25.958289 (+ 60us) service_pool.cc:225] Handling call
0114 03:04:25.958424 (+ 135us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:25.958484 (+ 60us) raft_consensus.cc:1529] Early marking committed up to index 42
0114 03:04:25.958554 (+ 70us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:25.958647 (+ 93us) write_op.cc:270] Start()
0114 03:04:26.158821 (+200174us) write_op.cc:276] Timestamp: P: 1736823864767446 usec, L: 0
0114 03:04:26.159121 (+ 300us) write_op.cc:270] Start()
0114 03:04:27.021317 (+862196us) write_op.cc:276] Timestamp: P: 1736823865430256 usec, L: 0
0114 03:04:27.021770 (+ 453us) log.cc:844] Serialized 708 byte log entry
0114 03:04:27.021871 (+ 101us) raft_consensus.cc:1640] Marking committed up to 42
0114 03:04:27.021879 (+ 8us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:27.021913 (+ 34us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:27.022402 (+ 489us) raft_consensus.cc:1680] finished
0114 03:04:27.022404 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:27.022447 (+ 43us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:26.159079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:26.159187 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:26.159191 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:26.159194 (+ 3us) tablet.cc:659] Decoding operations
0114 03:04:26.159262 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:26.159271 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:26.159274 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:26.159311 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:04:26.159312 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:04:27.021686 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:27.021797 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:27.021801 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:27.021803 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:27.021869 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:27.021878 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:27.021880 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:27.021918 (+ 38us) tablet.cc:698] Row locks acquired
0114 03:04:27.021922 (+ 4us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":51,"threads_started":1,"wal-append.queue_time_us":144,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":84,"prepare.run_cpu_time_us":323,"prepare.run_wall_time_us":324}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":230,"prepare.run_cpu_time_us":339,"prepare.run_wall_time_us":339,"thread_start_us":83,"threads_started":1}]]}
I20250114 03:04:27.023185 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:27.023195 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 44 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:27.023682 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:27.023849 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:27.458329 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:27.458488 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:27.459017 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:27.459478 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:27.459707 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 45 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:27.460180 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:27.460415 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:27.475824 4483 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=43, attempt_no=0}) took 1267 ms. Trace:
I20250114 03:04:27.475934 4483 rpcz_store.cc:276] 0114 03:04:26.208244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:26.208317 (+ 73us) service_pool.cc:225] Handling call
0114 03:04:27.475796 (+1267479us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:26.208726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:26.208861 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:26.208864 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:26.208865 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:26.208922 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:26.208928 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:26.208930 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:26.208962 (+ 32us) tablet.cc:698] Row locks acquired
0114 03:04:26.208963 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:26.209003 (+ 40us) write_op.cc:270] Start()
0114 03:04:26.209112 (+ 109us) write_op.cc:276] Timestamp: P: 1736823866208982 usec, L: 0
0114 03:04:26.209114 (+ 2us) op_driver.cc:348] REPLICATION: starting
0114 03:04:26.209286 (+ 172us) log.cc:844] Serialized 354 byte log entry
0114 03:04:27.475057 (+1265771us) op_driver.cc:464] REPLICATION: finished
0114 03:04:27.475348 (+ 291us) write_op.cc:301] APPLY: starting
0114 03:04:27.475370 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:27.475402 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:27.475420 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:27.475481 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:27.475506 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:27.475509 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:27.475583 (+ 74us) log.cc:844] Serialized 22 byte log entry
0114 03:04:27.475645 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:27.475662 (+ 17us) write_op.cc:454] Released schema lock
0114 03:04:27.475667 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":219,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":611,"replication_time_us":1265887,"thread_start_us":102,"threads_started":1}]]}
I20250114 03:04:28.058099 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:28.058252 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:28.058820 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:28.059175 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:28.059295 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:28.059733 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:28.071169 4490 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=44, attempt_no=0}) took 1341 ms. Trace:
I20250114 03:04:28.071269 4490 rpcz_store.cc:276] 0114 03:04:26.729760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:26.729819 (+ 59us) service_pool.cc:225] Handling call
0114 03:04:28.071141 (+1341322us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:26.730475 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:26.730620 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:26.730626 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:26.730629 (+ 3us) tablet.cc:659] Decoding operations
0114 03:04:26.730730 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:26.730744 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:26.730750 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:26.730802 (+ 52us) tablet.cc:698] Row locks acquired
0114 03:04:26.730804 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:26.730853 (+ 49us) write_op.cc:270] Start()
0114 03:04:26.852008 (+121155us) write_op.cc:276] Timestamp: P: 1736823866730835 usec, L: 0
0114 03:04:26.852011 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:26.852224 (+ 213us) log.cc:844] Serialized 354 byte log entry
0114 03:04:28.070403 (+1218179us) op_driver.cc:464] REPLICATION: finished
0114 03:04:28.070701 (+ 298us) write_op.cc:301] APPLY: starting
0114 03:04:28.070721 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:28.070751 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:28.070769 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:28.070832 (+ 63us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:28.070857 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:28.070861 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:28.070949 (+ 88us) log.cc:844] Serialized 22 byte log entry
0114 03:04:28.071020 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:28.071035 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:28.071042 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":226,"num_ops":1,"prepare.queue_time_us":207,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":121826,"replication_time_us":1218338,"thread_start_us":170,"threads_started":2}]]}
I20250114 03:04:28.340801 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:28.340979 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:28.341279 4198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:49933 (request call id 4662) took 1317 ms. Trace:
I20250114 03:04:28.341349 4198 rpcz_store.cc:276] 0114 03:04:27.023751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:27.023809 (+ 58us) service_pool.cc:225] Handling call
0114 03:04:27.023967 (+ 158us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:27.024049 (+ 82us) raft_consensus.cc:1529] Early marking committed up to index 44
0114 03:04:27.024284 (+ 235us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:27.024411 (+ 127us) write_op.cc:270] Start()
0114 03:04:27.586582 (+562171us) write_op.cc:276] Timestamp: P: 1736823866208982 usec, L: 0
0114 03:04:27.587015 (+ 433us) write_op.cc:270] Start()
0114 03:04:28.340180 (+753165us) write_op.cc:276] Timestamp: P: 1736823866730835 usec, L: 0
0114 03:04:28.340620 (+ 440us) log.cc:844] Serialized 708 byte log entry
0114 03:04:28.340714 (+ 94us) raft_consensus.cc:1640] Marking committed up to 44
0114 03:04:28.340721 (+ 7us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:28.340754 (+ 33us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:28.341213 (+ 459us) raft_consensus.cc:1680] finished
0114 03:04:28.341216 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:28.341265 (+ 49us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:27.587022 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:27.587130 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:27.587134 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:27.587136 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:27.587205 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:27.587214 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:27.587216 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:27.587252 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:04:27.587253 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:04:28.340552 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:28.340684 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:28.340689 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:28.340690 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:28.340798 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:28.340811 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:28.340814 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:28.340892 (+ 78us) tablet.cc:698] Row locks acquired
0114 03:04:28.340894 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":44,"threads_started":1,"wal-append.queue_time_us":158,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":268,"prepare.run_cpu_time_us":306,"prepare.run_wall_time_us":307,"thread_start_us":107,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":222,"prepare.run_cpu_time_us":431,"prepare.run_wall_time_us":431,"thread_start_us":83,"threads_started":1}]]}
I20250114 03:04:28.341602 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:28.342046 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:28.342139 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 46 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:28.342562 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:28.342754 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:28.672185 4158 tablet_service.cc:2970] error setting up scanner 7cef5a2622be46c5a8b4d14147718cbb with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.329s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:28.697021 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:28.707027 4498 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=45, attempt_no=0}) took 1229 ms. Trace:
I20250114 03:04:28.707134 4498 rpcz_store.cc:276] 0114 03:04:27.477374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:27.477434 (+ 60us) service_pool.cc:225] Handling call
0114 03:04:28.707003 (+1229569us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:27.478007 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:27.478100 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:27.478103 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:27.478104 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:27.478169 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:27.478179 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:27.478184 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:27.478224 (+ 40us) tablet.cc:698] Row locks acquired
0114 03:04:27.478226 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:27.478262 (+ 36us) write_op.cc:270] Start()
0114 03:04:27.777421 (+299159us) write_op.cc:276] Timestamp: P: 1736823867478221 usec, L: 0
0114 03:04:27.777424 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:27.777634 (+ 210us) log.cc:844] Serialized 354 byte log entry
0114 03:04:28.706260 (+928626us) op_driver.cc:464] REPLICATION: finished
0114 03:04:28.706552 (+ 292us) write_op.cc:301] APPLY: starting
0114 03:04:28.706572 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:28.706606 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:28.706624 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:28.706686 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:28.706713 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:28.706718 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:28.706793 (+ 75us) log.cc:844] Serialized 22 byte log entry
0114 03:04:28.706869 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:28.706885 (+ 16us) write_op.cc:454] Released schema lock
0114 03:04:28.706890 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":215,"num_ops":1,"prepare.queue_time_us":179,"prepare.run_cpu_time_us":622,"prepare.run_wall_time_us":299687,"replication_time_us":928784,"thread_start_us":174,"threads_started":2}]]}
I20250114 03:04:29.002633 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:29.002832 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:29.003325 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:29.003741 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:29.003773 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:29.004254 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:29.195829 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:29.196004 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:29.196617 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:29.197033 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:29.197062 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 48 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:29.197502 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
W20250114 03:04:29.351003 4159 tablet_service.cc:2970] error setting up scanner ddacfb72fa9d44bbace35564f5f467b6 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823868350752 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823868073374 usec, L: 0 Physical time difference: 0.277s: new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
I20250114 03:04:29.351392 4159 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Scan from 127.0.0.1:52220 (request call id 26) took 1001 ms. Trace:
I20250114 03:04:29.351445 4159 rpcz_store.cc:276] 0114 03:04:28.350301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:28.350363 (+ 62us) service_pool.cc:225] Handling call
0114 03:04:28.350533 (+ 170us) tablet_service.cc:2780] Created scanner ddacfb72fa9d44bbace35564f5f467b6 for tablet d5bcdea4a70943628babc9dfd8c94f1e, query id is ee819643cb794686999f6fe006fdff21
0114 03:04:28.350730 (+ 197us) tablet_service.cc:2920] Creating iterator
0114 03:04:28.350790 (+ 60us) tablet_service.cc:3297] Waiting safe time to advance
0114 03:04:29.350973 (+1000183us) tablet_service.cc:2945] Iterator created
0114 03:04:29.350990 (+ 17us) tablet_service.cc:2967] Iterator init: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: Timed out waiting for ts: P: 1736823868350752 usec, L: 0 to be safe (mode: NON-LEADER). Current safe time: P: 1736823868073374 usec, L: 0 Physical time difference: 0.277s
0114 03:04:29.351382 (+ 392us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250114 03:04:29.875325 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:29.899350 4508 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=47, attempt_no=0}) took 1190 ms. Trace:
I20250114 03:04:29.899461 4508 rpcz_store.cc:276] 0114 03:04:28.708467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:28.708532 (+ 65us) service_pool.cc:225] Handling call
0114 03:04:29.899324 (+1190792us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:28.709109 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:28.709203 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:28.709206 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:28.709207 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:28.709265 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:28.709271 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:28.709273 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:28.709306 (+ 33us) tablet.cc:698] Row locks acquired
0114 03:04:28.709307 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:28.709333 (+ 26us) write_op.cc:270] Start()
0114 03:04:28.715466 (+ 6133us) write_op.cc:276] Timestamp: P: 1736823868709264 usec, L: 0
0114 03:04:28.715470 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:28.715666 (+ 196us) log.cc:844] Serialized 354 byte log entry
0114 03:04:29.898589 (+1182923us) op_driver.cc:464] REPLICATION: finished
0114 03:04:29.898893 (+ 304us) write_op.cc:301] APPLY: starting
0114 03:04:29.898914 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:29.898945 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:29.898962 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:29.899041 (+ 79us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:29.899065 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:29.899069 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:29.899142 (+ 73us) log.cc:844] Serialized 22 byte log entry
0114 03:04:29.899206 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:29.899222 (+ 16us) write_op.cc:454] Released schema lock
0114 03:04:29.899228 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":233,"num_ops":1,"prepare.queue_time_us":158,"prepare.run_cpu_time_us":551,"prepare.run_wall_time_us":6614,"replication_time_us":1183064,"thread_start_us":198,"threads_started":2}]]}
I20250114 03:04:30.657166 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:30.657315 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:30.657850 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:30.658216 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:30.658285 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 50 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:30.658860 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:30.678380 4515 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=48, attempt_no=0}) took 1640 ms. Trace:
I20250114 03:04:30.678476 4515 rpcz_store.cc:276] 0114 03:04:29.037537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:29.037619 (+ 82us) service_pool.cc:225] Handling call
0114 03:04:30.678359 (+1640740us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:29.038098 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:29.038192 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:29.038196 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:29.038197 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:29.038254 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:29.038261 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:29.038264 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:29.038294 (+ 30us) tablet.cc:698] Row locks acquired
0114 03:04:29.038295 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:29.038324 (+ 29us) write_op.cc:270] Start()
0114 03:04:29.179507 (+141183us) write_op.cc:276] Timestamp: P: 1736823869038282 usec, L: 0
0114 03:04:29.179511 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:29.179759 (+ 248us) log.cc:844] Serialized 354 byte log entry
0114 03:04:30.677651 (+1497892us) op_driver.cc:464] REPLICATION: finished
0114 03:04:30.677982 (+ 331us) write_op.cc:301] APPLY: starting
0114 03:04:30.678003 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:30.678033 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:30.678052 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:30.678114 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:30.678137 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:30.678140 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:30.678213 (+ 73us) log.cc:844] Serialized 22 byte log entry
0114 03:04:30.678265 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:30.678278 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:30.678283 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":257,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":643,"prepare.run_wall_time_us":141727,"replication_time_us":1498066,"thread_start_us":95,"threads_started":1}]]}
I20250114 03:04:31.098652 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:31.099161 4198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.65:49933 (request call id 4664) took 1901 ms. Trace:
I20250114 03:04:31.099242 4198 rpcz_store.cc:276] 0114 03:04:29.197416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:29.197482 (+ 66us) service_pool.cc:225] Handling call
0114 03:04:29.197646 (+ 164us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:29.197737 (+ 91us) raft_consensus.cc:1529] Early marking committed up to index 48
0114 03:04:29.197985 (+ 248us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:29.198147 (+ 162us) write_op.cc:270] Start()
0114 03:04:30.197304 (+999157us) write_op.cc:276] Timestamp: P: 1736823868709264 usec, L: 0
0114 03:04:30.197709 (+ 405us) write_op.cc:270] Start()
0114 03:04:31.097947 (+900238us) write_op.cc:276] Timestamp: P: 1736823869038282 usec, L: 0
0114 03:04:31.098413 (+ 466us) log.cc:844] Serialized 708 byte log entry
0114 03:04:31.098519 (+ 106us) raft_consensus.cc:1640] Marking committed up to 48
0114 03:04:31.098526 (+ 7us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:31.098561 (+ 35us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:31.099072 (+ 511us) raft_consensus.cc:1680] finished
0114 03:04:31.099074 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:31.099125 (+ 51us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:30.197706 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:30.197816 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:30.197820 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:30.197821 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:30.197889 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:30.197900 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:30.197902 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:30.197941 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:04:30.197942 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:04:31.098356 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:31.098460 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:31.098464 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:31.098466 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:31.098540 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:31.098550 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:31.098552 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:31.098632 (+ 80us) tablet.cc:698] Row locks acquired
0114 03:04:31.098634 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":54,"threads_started":1,"wal-append.queue_time_us":139,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":240,"prepare.run_cpu_time_us":312,"prepare.run_wall_time_us":312,"thread_start_us":89,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":258,"prepare.run_cpu_time_us":366,"prepare.run_wall_time_us":365,"thread_start_us":103,"threads_started":1}]]}
I20250114 03:04:31.159052 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:31.177819 4515 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=49, attempt_no=0}) took 1277 ms. Trace:
I20250114 03:04:31.177918 4515 rpcz_store.cc:276] 0114 03:04:29.900854 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:29.900895 (+ 41us) service_pool.cc:225] Handling call
0114 03:04:31.177797 (+1276902us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:29.901481 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:29.901576 (+ 95us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:29.901579 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:29.901580 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:29.901640 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:29.901647 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:29.901648 (+ 1us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:29.901682 (+ 34us) tablet.cc:698] Row locks acquired
0114 03:04:29.901683 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:29.901712 (+ 29us) write_op.cc:270] Start()
0114 03:04:29.922906 (+ 21194us) write_op.cc:276] Timestamp: P: 1736823869901683 usec, L: 0
0114 03:04:29.922911 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:29.923145 (+ 234us) log.cc:844] Serialized 354 byte log entry
0114 03:04:31.177201 (+1254056us) op_driver.cc:464] REPLICATION: finished
0114 03:04:31.177362 (+ 161us) write_op.cc:301] APPLY: starting
0114 03:04:31.177386 (+ 24us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:31.177418 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:31.177432 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:31.177490 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:31.177518 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:31.177521 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:31.177596 (+ 75us) log.cc:844] Serialized 22 byte log entry
0114 03:04:31.177659 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:31.177674 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:31.177679 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":64,"num_ops":1,"prepare.queue_time_us":200,"prepare.run_cpu_time_us":640,"prepare.run_wall_time_us":21732,"replication_time_us":1254252,"thread_start_us":90,"threads_started":1}]]}
I20250114 03:04:31.617887 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:31.618070 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:31.618592 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:31.619014 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:31.619074 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 51 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:31.619572 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:32.006510 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:32.055550 4527 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=50, attempt_no=0}) took 1375 ms. Trace:
I20250114 03:04:32.055649 4527 rpcz_store.cc:276] 0114 03:04:30.679794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:30.679855 (+ 61us) service_pool.cc:225] Handling call
0114 03:04:32.055528 (+1375673us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:30.680346 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:30.680433 (+ 87us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:30.680435 (+ 2us) write_op.cc:435] Acquired schema lock
0114 03:04:30.680436 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:30.680492 (+ 56us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:30.680498 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:30.680500 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:30.680531 (+ 31us) tablet.cc:698] Row locks acquired
0114 03:04:30.680532 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:30.680558 (+ 26us) write_op.cc:270] Start()
0114 03:04:31.204697 (+524139us) write_op.cc:276] Timestamp: P: 1736823870680500 usec, L: 0
0114 03:04:31.204701 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:31.204913 (+ 212us) log.cc:844] Serialized 354 byte log entry
0114 03:04:32.054725 (+849812us) op_driver.cc:464] REPLICATION: finished
0114 03:04:32.055070 (+ 345us) write_op.cc:301] APPLY: starting
0114 03:04:32.055090 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:32.055121 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:32.055139 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:32.055202 (+ 63us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:32.055228 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:32.055231 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:32.055304 (+ 73us) log.cc:844] Serialized 22 byte log entry
0114 03:04:32.055409 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:32.055426 (+ 17us) write_op.cc:454] Released schema lock
0114 03:04:32.055432 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":263,"num_ops":1,"prepare.queue_time_us":132,"prepare.run_cpu_time_us":571,"prepare.run_wall_time_us":524637,"raft.queue_time_us":76,"raft.run_cpu_time_us":38,"raft.run_wall_time_us":38,"replication_time_us":849962,"spinlock_wait_cycles":62848,"thread_start_us":175,"threads_started":2}]]}
I20250114 03:04:32.191475 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:32.191648 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:32.192256 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:32.192633 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:32.192670 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 52 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:32.193221 4003 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
W20250114 03:04:32.681509 4159 tablet_service.cc:2970] error setting up scanner 6180693f9d3249449bd8db07950d99be with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.673s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "ee819643cb794686999f6fe006fdff21"
W20250114 03:04:32.770179 4159 tablet_service.cc:2970] error setting up scanner dce93bb1771f4f75aa604cd8773d8773 with request: Service unavailable: Timed out: could not wait for desired snapshot timestamp to be consistent: tablet hasn't heard from leader or there hasn't been a stable leader for 0.762s (maximum allowed 0.100s): new_scan_request { tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" projected_columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } projected_columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } projected_columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } read_mode: READ_AT_SNAPSHOT cache_blocks: true order_mode: ORDERED row_format_flags: 0 authz_token { token_data: "\010\316\252\227\274\006\"3\n\005slave\022*\n 0d0c42f365e64bc1bd0760b81d32cf5d\020\001\030\001 \001(\001" signature: "<redacted>"",\023\035\017cD\007]\023}\3459\243\323\2502\021]\237ed\235\244\371\014\274\341\t\207\017\265\257\320M\3119Cih\025\303\211\206{\376\023\004\276\266\323\207W\234*\332\"\264,\316\376?j\300-" signing_key_seq_num: 0 } } call_seq_id: 0 query_id: "f29e5d225b314cab9957bc0e7a2d8e4e"
I20250114 03:04:32.894410 4368 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:32.894551 4368 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:32.895079 4368 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:32.895478 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 53 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:32.895607 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 53 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:32.896083 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:32.912235 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:32.941629 4537 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=51, attempt_no=0}) took 1762 ms. Trace:
I20250114 03:04:32.941726 4537 rpcz_store.cc:276] 0114 03:04:31.179181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:31.179240 (+ 59us) service_pool.cc:225] Handling call
0114 03:04:32.941602 (+1762362us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:31.205030 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:31.205143 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:31.205148 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:31.205149 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:31.205222 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:31.205232 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:31.205235 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:31.205278 (+ 43us) tablet.cc:698] Row locks acquired
0114 03:04:31.205279 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:31.205325 (+ 46us) write_op.cc:270] Start()
0114 03:04:32.034476 (+829151us) write_op.cc:276] Timestamp: P: 1736823871205261 usec, L: 0
0114 03:04:32.034479 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:32.034652 (+ 173us) log.cc:844] Serialized 354 byte log entry
0114 03:04:32.940884 (+906232us) op_driver.cc:464] REPLICATION: finished
0114 03:04:32.941187 (+ 303us) write_op.cc:301] APPLY: starting
0114 03:04:32.941209 (+ 22us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:32.941239 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:32.941258 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:32.941324 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:32.941350 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:32.941353 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:32.941427 (+ 74us) log.cc:844] Serialized 22 byte log entry
0114 03:04:32.941482 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:32.941495 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:32.941500 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":228,"num_ops":1,"prepare.queue_time_us":25391,"prepare.run_cpu_time_us":599,"prepare.run_wall_time_us":829683,"replication_time_us":906315,"thread_start_us":84,"threads_started":1}]]}
I20250114 03:04:33.074175 4368 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:33.075752 4537 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:33370 (ReqId={client: 1156095772d54fcc9f5959849ac6df4f, seq_no=52, attempt_no=0}) took 1018 ms. Trace:
I20250114 03:04:33.075865 4537 rpcz_store.cc:276] 0114 03:04:32.057208 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:32.057276 (+ 68us) service_pool.cc:225] Handling call
0114 03:04:33.075733 (+1018457us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:32.057778 (+ 0us) write_op.cc:183] PREPARE: starting on tablet d5bcdea4a70943628babc9dfd8c94f1e
0114 03:04:32.057883 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:32.057887 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:32.057888 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:32.057964 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:32.057971 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:32.057973 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:32.058006 (+ 33us) tablet.cc:698] Row locks acquired
0114 03:04:32.058007 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:32.058041 (+ 34us) write_op.cc:270] Start()
0114 03:04:32.243200 (+185159us) write_op.cc:276] Timestamp: P: 1736823872057989 usec, L: 0
0114 03:04:32.243204 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:32.243416 (+ 212us) log.cc:844] Serialized 354 byte log entry
0114 03:04:33.075151 (+831735us) op_driver.cc:464] REPLICATION: finished
0114 03:04:33.075256 (+ 105us) write_op.cc:301] APPLY: starting
0114 03:04:33.075274 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:33.075303 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:33.075325 (+ 22us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:33.075391 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:33.075421 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:33.075425 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:33.075483 (+ 58us) log.cc:844] Serialized 22 byte log entry
0114 03:04:33.075542 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:33.075586 (+ 44us) write_op.cc:454] Released schema lock
0114 03:04:33.075590 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":43,"num_ops":1,"prepare.queue_time_us":87,"prepare.run_cpu_time_us":652,"prepare.run_wall_time_us":185713,"replication_time_us":831905}]]}
I20250114 03:04:33.254590 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:33.254730 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:33.255362 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:33.255702 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:33.255769 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:33.256246 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:33.256446 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:33.389561 4451 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:33.389698 4451 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:33.390086 4451 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:33.390393 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:33.390412 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:33.390954 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:33.391131 4451 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:34.932581 4538 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:34.932706 4538 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:34.933342 4538 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:34.933761 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:34.933799 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:34.934298 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:34.934458 4538 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:35.248438 4293 scanner-internal.cc:458] Time spent opening tablet: real 32.429s user 0.008s sys 0.000s
I20250114 03:04:35.389117 4538 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:35.389253 4538 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:35.389672 4538 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:35.389974 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:35.390017 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:35.390456 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:35.390652 4538 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250114 03:04:36.359586 4292 scanner-internal.cc:458] Time spent opening tablet: real 33.543s user 0.007s sys 0.000s
I20250114 03:04:36.389886 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:36.390027 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.390791 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.390940 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:36.392405 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.392868 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:36.392946 4549 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:36.393039 4548 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:36.393045 4549 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.393440 4549 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:36.394083 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.394095 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:36.394731 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:36.394955 4549 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:36.430626 3903 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:04:36.430649 4047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:04:36.431569 4179 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:04:36.442356 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:36.442505 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.442922 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.443365 4549 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:36.443477 4549 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.443833 4549 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:36.444001 4064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:36.444108 4064 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f21921611b433b9bbf44c95152b278 in term 1.
I20250114 03:04:36.444255 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:36.444425 4133 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f, e1f21921611b433b9bbf44c95152b278; no voters:
I20250114 03:04:36.444619 4548 raft_consensus.cc:2798] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 03:04:36.444674 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:36.444657 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.444726 4548 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:04:36.446018 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.446236 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.446424 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 election: Requested vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.446780 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f"
I20250114 03:04:36.447016 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420"
I20250114 03:04:36.446976 4532 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.447520 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.447744 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:36.448423 4548 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 03:04:36.448549 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.448758 4000 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ac0078d577b48d8b07432520bba2e4f; no voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278
I20250114 03:04:36.449230 4533 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.449481 4549 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:04:36.450867 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.451406 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.453191 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.453811 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.455319 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.455833 4533 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.457427 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:36.457530 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.457705 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.458237 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.458262 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.458428 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:36.458534 4533 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.458989 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:36.459138 4548 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 03:04:36.460065 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.460431 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.462332 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.462761 4569 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
bcc99ceacac04f819c6bc0b05b700fa3 | 127.1.109.126:34647 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:37681 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3a063722f0224c6bbbcfb6b63ce83420 | 127.1.109.65:40781 | HEALTHY | <none> | 1 | 0
8ac0078d577b48d8b07432520bba2e4f | 127.1.109.66:35297 | HEALTHY | <none> | 0 | 0
e1f21921611b433b9bbf44c95152b278 | 127.1.109.67:39007 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
consensus_inject_latency_ms_in_notifications | 50 | unsafe,hidden | all 3 server(s) checked
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:40781
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:35297
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:39007
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:40781
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:35297
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:39007
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:37681 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
I20250114 03:04:36.464433 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250114 03:04:36.464849 4569 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.466807 1461 log_verifier.cc:126] Checking tablet d5bcdea4a70943628babc9dfd8c94f1e
I20250114 03:04:36.466938 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.467237 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:36.467344 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.467763 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.468237 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:36.468533 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.470077 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.470438 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.470463 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e1f21921611b433b9bbf44c95152b278; no voters: 3a063722f0224c6bbbcfb6b63ce83420, 8ac0078d577b48d8b07432520bba2e4f
I20250114 03:04:36.470785 4548 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 03:04:36.471128 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.472971 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.473407 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.475164 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.475569 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.477360 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.477864 4569 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.479733 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.480154 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.481914 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.482335 4569 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.484215 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.484686 4533 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.486709 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.487165 4533 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.488662 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.489176 4540 consensus_queue.cc:1046] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 }, Status: INVALID_TERM, Last received: 1.54, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.490548 4532 raft_consensus.cc:3049] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:04:36.490636 4532 raft_consensus.cc:738] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3a063722f0224c6bbbcfb6b63ce83420, State: Running, Role: LEADER
I20250114 03:04:36.490847 4532 consensus_queue.cc:260] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.491008 4532 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:04:36.492472 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:36.492591 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.492852 4199 raft_consensus.cc:1235] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Rejecting Update request from peer 3a063722f0224c6bbbcfb6b63ce83420 for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:36.493001 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.493330 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.493448 3935 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1f21921611b433b9bbf44c95152b278 in term 2.
I20250114 03:04:36.493678 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f" is_pre_election: true
I20250114 03:04:36.493731 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278; no voters:
I20250114 03:04:36.493950 4548 raft_consensus.cc:2798] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 03:04:36.494005 4548 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:36.494055 4548 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:04:36.494923 4548 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.495232 4548 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 election: Requested vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), 8ac0078d577b48d8b07432520bba2e4f (127.1.109.66:35297)
I20250114 03:04:36.495710 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420"
I20250114 03:04:36.495810 3935 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:04:36.496551 3935 raft_consensus.cc:2463] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1f21921611b433b9bbf44c95152b278 in term 3.
I20250114 03:04:36.495565 4067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "e1f21921611b433b9bbf44c95152b278" candidate_term: 3 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "8ac0078d577b48d8b07432520bba2e4f"
I20250114 03:04:36.496866 4132 leader_election.cc:304] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a063722f0224c6bbbcfb6b63ce83420, e1f21921611b433b9bbf44c95152b278; no voters:
I20250114 03:04:36.497033 4548 raft_consensus.cc:2798] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:04:36.497218 4548 raft_consensus.cc:695] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [term 3 LEADER]: Becoming Leader. State: Replica: e1f21921611b433b9bbf44c95152b278, State: Running, Role: LEADER
I20250114 03:04:36.497524 4548 consensus_queue.cc:237] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 54, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.497697 4549 raft_consensus.cc:491] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3a063722f0224c6bbbcfb6b63ce83420)
I20250114 03:04:36.497814 4549 raft_consensus.cc:513] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } }
I20250114 03:04:36.498247 4549 leader_election.cc:290] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781), e1f21921611b433b9bbf44c95152b278 (127.1.109.67:39007)
I20250114 03:04:36.498693 3935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "3a063722f0224c6bbbcfb6b63ce83420" is_pre_election: true
I20250114 03:04:36.498836 3935 raft_consensus.cc:2363] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ac0078d577b48d8b07432520bba2e4f for earlier term 2. Current term is 3.
I20250114 03:04:36.498831 4199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5bcdea4a70943628babc9dfd8c94f1e" candidate_uuid: "8ac0078d577b48d8b07432520bba2e4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 54 } } ignore_live_leader: false dest_uuid: "e1f21921611b433b9bbf44c95152b278" is_pre_election: true
I20250114 03:04:36.499470 4000 leader_election.cc:400] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Vote denied by peer 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781) with higher term. Message: Invalid argument: T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ac0078d577b48d8b07432520bba2e4f for earlier term 2. Current term is 3.
I20250114 03:04:36.499524 4000 leader_election.cc:403] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 03:04:36.500072 3792 catalog_manager.cc:5526] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 reported cstate change: term changed from 1 to 3, leader changed from 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65) to e1f21921611b433b9bbf44c95152b278 (127.1.109.67). New cstate: current_term: 3 leader_uuid: "e1f21921611b433b9bbf44c95152b278" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e1f21921611b433b9bbf44c95152b278" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39007 } health_report { overall_health: HEALTHY } } }
I20250114 03:04:36.500285 3935 raft_consensus.cc:1270] T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 3 FOLLOWER]: Refusing update from remote peer e1f21921611b433b9bbf44c95152b278: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 3 index: 55. (index mismatch)
I20250114 03:04:36.500746 4067 raft_consensus.cc:3054] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 1 FOLLOWER]: Advancing to term 3
I20250114 03:04:36.500978 4548 consensus_queue.cc:1035] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a063722f0224c6bbbcfb6b63ce83420" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.501835 4067 raft_consensus.cc:1270] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 3 FOLLOWER]: Refusing update from remote peer e1f21921611b433b9bbf44c95152b278: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 3 index: 55. (index mismatch)
I20250114 03:04:36.501946 4549 raft_consensus.cc:2743] T d5bcdea4a70943628babc9dfd8c94f1e P 8ac0078d577b48d8b07432520bba2e4f [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781) with higher term. Message: Invalid argument: T d5bcdea4a70943628babc9dfd8c94f1e P 3a063722f0224c6bbbcfb6b63ce83420 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8ac0078d577b48d8b07432520bba2e4f for earlier term 2. Current term is 3.
I20250114 03:04:36.502457 4570 consensus_queue.cc:1035] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ac0078d577b48d8b07432520bba2e4f" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20250114 03:04:36.508368 1461 log_verifier.cc:177] Verified matching terms for 54 ops in tablet d5bcdea4a70943628babc9dfd8c94f1e
I20250114 03:04:36.528313 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 3853
W20250114 03:04:36.536837 4132 connection.cc:537] client connection to 127.1.109.65:40781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:04:36.536971 4132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20250114 03:04:36.537710 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 3985
W20250114 03:04:36.545217 4132 consensus_peers.cc:487] T d5bcdea4a70943628babc9dfd8c94f1e P e1f21921611b433b9bbf44c95152b278 -> Peer 3a063722f0224c6bbbcfb6b63ce83420 (127.1.109.65:40781): Couldn't send request to peer 3a063722f0224c6bbbcfb6b63ce83420. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.65:40781: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 03:04:36.546480 4133 connection.cc:537] client connection to 127.1.109.66:35297 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 03:04:36.547224 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 4117
I20250114 03:04:36.555116 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 3761
2025-01-14T03:04:36Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithNotificationLatency (34863 ms)
[ RUN ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys
I20250114 03:04:36.579728 1461 test_util.cc:274] Using random seed: -768380544
I20250114 03:04:36.580531 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:04:36.580578 1461 ts_itest-base.cc:113] --------------
I20250114 03:04:36.580600 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:04:36.580617 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:04:36.580644 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:04:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:04:36Z Disabled control of system clock
I20250114 03:04:36.589159 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34547
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:34547 with env {}
W20250114 03:04:36.700357 4598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:36.700629 4598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:36.700707 4598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:36.704061 4598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:04:36.704165 4598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:36.704198 4598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:04:36.704228 4598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:04:36.708269 4598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:34547
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.4598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:36.709486 4598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:36.710616 4598 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:36.718779 4604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:36.719187 4606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:36.720180 4603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:36.721071 4598 server_base.cc:1034] running on GCE node
I20250114 03:04:36.721668 4598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:36.722729 4598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:36.723902 4598 hybrid_clock.cc:648] HybridClock initialized: now 1736823876723870 us; error 44 us; skew 500 ppm
I20250114 03:04:36.726084 4598 webserver.cc:458] Webserver started at http://127.1.109.126:39545/ using document root <none> and password file <none>
I20250114 03:04:36.726699 4598 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:36.726765 4598 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:36.726950 4598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:36.728760 4598 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "aa45548db049458cb0ae92a9bdf3d484"
format_stamp: "Formatted at 2025-01-14 03:04:36 on dist-test-slave-r16g"
I20250114 03:04:36.729329 4598 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "aa45548db049458cb0ae92a9bdf3d484"
format_stamp: "Formatted at 2025-01-14 03:04:36 on dist-test-slave-r16g"
I20250114 03:04:36.733063 4598 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250114 03:04:36.735467 4612 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:36.736610 4598 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:04:36.736740 4598 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "aa45548db049458cb0ae92a9bdf3d484"
format_stamp: "Formatted at 2025-01-14 03:04:36 on dist-test-slave-r16g"
I20250114 03:04:36.736873 4598 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:36.774063 4598 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:36.774787 4598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:36.774958 4598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:36.781803 4664 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:34547 every 8 connection(s)
I20250114 03:04:36.781807 4598 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:34547
I20250114 03:04:36.782853 4598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:04:36.785348 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 4598
I20250114 03:04:36.785444 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:04:36.785604 4665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:36.791601 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484: Bootstrap starting.
I20250114 03:04:36.794123 4665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:36.795115 4665 log.cc:826] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:36.797068 4665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484: No bootstrap required, opened a new log
I20250114 03:04:36.800272 4665 raft_consensus.cc:357] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa45548db049458cb0ae92a9bdf3d484" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34547 } }
I20250114 03:04:36.800498 4665 raft_consensus.cc:383] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:36.800536 4665 raft_consensus.cc:738] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa45548db049458cb0ae92a9bdf3d484, State: Initialized, Role: FOLLOWER
I20250114 03:04:36.800980 4665 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa45548db049458cb0ae92a9bdf3d484" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34547 } }
I20250114 03:04:36.801112 4665 raft_consensus.cc:397] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:04:36.801159 4665 raft_consensus.cc:491] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:04:36.801242 4665 raft_consensus.cc:3054] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:36.802044 4665 raft_consensus.cc:513] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa45548db049458cb0ae92a9bdf3d484" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34547 } }
I20250114 03:04:36.802349 4665 leader_election.cc:304] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa45548db049458cb0ae92a9bdf3d484; no voters:
I20250114 03:04:36.802613 4665 leader_election.cc:290] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:04:36.802776 4670 raft_consensus.cc:2798] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:04:36.803012 4670 raft_consensus.cc:695] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [term 1 LEADER]: Becoming Leader. State: Replica: aa45548db049458cb0ae92a9bdf3d484, State: Running, Role: LEADER
I20250114 03:04:36.803351 4670 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa45548db049458cb0ae92a9bdf3d484" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34547 } }
I20250114 03:04:36.803766 4665 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:04:36.805238 4672 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa45548db049458cb0ae92a9bdf3d484. Latest consensus state: current_term: 1 leader_uuid: "aa45548db049458cb0ae92a9bdf3d484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa45548db049458cb0ae92a9bdf3d484" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34547 } } }
I20250114 03:04:36.805303 4671 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa45548db049458cb0ae92a9bdf3d484" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa45548db049458cb0ae92a9bdf3d484" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34547 } } }
I20250114 03:04:36.805377 4672 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [sys.catalog]: This master's current role is: LEADER
I20250114 03:04:36.805402 4671 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484 [sys.catalog]: This master's current role is: LEADER
I20250114 03:04:36.805795 4679 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:04:36.808951 4679 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:04:36.814483 4679 catalog_manager.cc:1348] Generated new cluster ID: 748280ded98749659919af966fdba183
I20250114 03:04:36.814596 4679 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:04:36.834841 4679 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:04:36.836179 4679 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:04:36.848037 4679 catalog_manager.cc:5899] T 00000000000000000000000000000000 P aa45548db049458cb0ae92a9bdf3d484: Generated new TSK 0
I20250114 03:04:36.848982 4679 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:04:36.859819 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34547
--builtin_ntp_servers=127.1.109.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20250114 03:04:36.973459 4689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:36.973690 4689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:36.973749 4689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:36.977064 4689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:36.977185 4689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:04:36.981067 4689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:34547
--never_fsync=true
--heap_profile_path=/tmp/kudu.4689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:36.982218 4689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:36.983364 4689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:36.992484 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:36.993093 4694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:36.994149 4695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:36.995029 4689 server_base.cc:1034] running on GCE node
I20250114 03:04:36.995460 4689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:36.996121 4689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:36.997300 4689 hybrid_clock.cc:648] HybridClock initialized: now 1736823876997254 us; error 44 us; skew 500 ppm
I20250114 03:04:36.999392 4689 webserver.cc:458] Webserver started at http://127.1.109.65:46589/ using document root <none> and password file <none>
I20250114 03:04:37.000001 4689 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:37.000066 4689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:37.000275 4689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:37.002079 4689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f6f5a4e02dd1463db75006734f788a32"
format_stamp: "Formatted at 2025-01-14 03:04:36 on dist-test-slave-r16g"
I20250114 03:04:37.002578 4689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f6f5a4e02dd1463db75006734f788a32"
format_stamp: "Formatted at 2025-01-14 03:04:36 on dist-test-slave-r16g"
I20250114 03:04:37.006122 4689 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:04:37.008368 4703 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.009496 4689 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:04:37.009627 4689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f6f5a4e02dd1463db75006734f788a32"
format_stamp: "Formatted at 2025-01-14 03:04:36 on dist-test-slave-r16g"
I20250114 03:04:37.009717 4689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:37.020304 4689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:37.021109 4689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:37.021291 4689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:37.021864 4689 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:04:37.022807 4689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:04:37.022859 4689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.022925 4689 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:04:37.022953 4689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.032058 4689 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:37273
I20250114 03:04:37.032083 4816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:37273 every 8 connection(s)
I20250114 03:04:37.033110 4689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:04:37.035856 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 4689
I20250114 03:04:37.035970 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:04:37.038643 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34547
--builtin_ntp_servers=127.1.109.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
I20250114 03:04:37.043136 4817 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34547
I20250114 03:04:37.043365 4817 heartbeater.cc:463] Registering TS with master...
I20250114 03:04:37.043920 4817 heartbeater.cc:510] Master 127.1.109.126:34547 requested a full tablet report, sending...
I20250114 03:04:37.045655 4629 ts_manager.cc:194] Registered new tserver with Master: f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273)
I20250114 03:04:37.047430 4629 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:56141
W20250114 03:04:37.150482 4820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:37.150718 4820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:37.150766 4820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:37.153970 4820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:37.154079 4820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:04:37.157954 4820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:34547
--never_fsync=true
--heap_profile_path=/tmp/kudu.4820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:37.159126 4820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:37.160261 4820 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:37.169184 4826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:37.170089 4828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:37.170804 4825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:37.171917 4820 server_base.cc:1034] running on GCE node
I20250114 03:04:37.172346 4820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:37.173007 4820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:37.174165 4820 hybrid_clock.cc:648] HybridClock initialized: now 1736823877174152 us; error 26 us; skew 500 ppm
I20250114 03:04:37.176086 4820 webserver.cc:458] Webserver started at http://127.1.109.66:32869/ using document root <none> and password file <none>
I20250114 03:04:37.176676 4820 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:37.176740 4820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:37.177065 4820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:37.178880 4820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "77a32f4dd261435cb6dc91526239a9cd"
format_stamp: "Formatted at 2025-01-14 03:04:37 on dist-test-slave-r16g"
I20250114 03:04:37.179378 4820 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "77a32f4dd261435cb6dc91526239a9cd"
format_stamp: "Formatted at 2025-01-14 03:04:37 on dist-test-slave-r16g"
I20250114 03:04:37.182960 4820 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:04:37.185284 4834 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.186376 4820 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:04:37.186489 4820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "77a32f4dd261435cb6dc91526239a9cd"
format_stamp: "Formatted at 2025-01-14 03:04:37 on dist-test-slave-r16g"
I20250114 03:04:37.186590 4820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:37.203018 4820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:37.203703 4820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:37.203868 4820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:37.204474 4820 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:04:37.205489 4820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:04:37.205542 4820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.205611 4820 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:04:37.205637 4820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.214457 4820 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:37591
I20250114 03:04:37.214481 4947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:37591 every 8 connection(s)
I20250114 03:04:37.215472 4820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:04:37.224504 4948 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34547
I20250114 03:04:37.224769 4948 heartbeater.cc:463] Registering TS with master...
I20250114 03:04:37.224927 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 4820
I20250114 03:04:37.225049 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:04:37.225526 4948 heartbeater.cc:510] Master 127.1.109.126:34547 requested a full tablet report, sending...
I20250114 03:04:37.226565 4628 ts_manager.cc:194] Registered new tserver with Master: 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.227314 4628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:47213
I20250114 03:04:37.227432 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34547
--builtin_ntp_servers=127.1.109.84:43617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--inject_latency_ms_before_starting_op=1000 with env {}
W20250114 03:04:37.338960 4951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:04:37.339188 4951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:04:37.339236 4951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:04:37.342442 4951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:04:37.342562 4951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:04:37.346396 4951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=2
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--inject_latency_ms_before_starting_op=1000
--tserver_master_addrs=127.1.109.126:34547
--never_fsync=true
--heap_profile_path=/tmp/kudu.4951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:04:37.347527 4951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:04:37.348664 4951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:04:37.357721 4959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:37.358417 4956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:04:37.359350 4957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:04:37.360293 4951 server_base.cc:1034] running on GCE node
I20250114 03:04:37.360719 4951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:04:37.361361 4951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:04:37.362516 4951 hybrid_clock.cc:648] HybridClock initialized: now 1736823877362500 us; error 31 us; skew 500 ppm
I20250114 03:04:37.364514 4951 webserver.cc:458] Webserver started at http://127.1.109.67:36697/ using document root <none> and password file <none>
I20250114 03:04:37.365146 4951 fs_manager.cc:362] Metadata directory not provided
I20250114 03:04:37.365206 4951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:04:37.365401 4951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:04:37.367519 4951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "879969dde1ff460bb29cf9319dcd8746"
format_stamp: "Formatted at 2025-01-14 03:04:37 on dist-test-slave-r16g"
I20250114 03:04:37.368023 4951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "879969dde1ff460bb29cf9319dcd8746"
format_stamp: "Formatted at 2025-01-14 03:04:37 on dist-test-slave-r16g"
I20250114 03:04:37.371579 4951 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20250114 03:04:37.373831 4965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.374876 4951 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:04:37.374986 4951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "879969dde1ff460bb29cf9319dcd8746"
format_stamp: "Formatted at 2025-01-14 03:04:37 on dist-test-slave-r16g"
I20250114 03:04:37.375075 4951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:04:37.386730 4951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:04:37.387364 4951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:04:37.387530 4951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:04:37.388089 4951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:04:37.389077 4951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:04:37.389127 4951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.389186 4951 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:04:37.389211 4951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:04:37.397686 4951 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:44225
I20250114 03:04:37.397737 5079 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:44225 every 8 connection(s)
I20250114 03:04:37.398672 4951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:04:37.403491 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 4951
I20250114 03:04:37.403591 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:04:37.407686 5081 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34547
I20250114 03:04:37.407930 5081 heartbeater.cc:463] Registering TS with master...
I20250114 03:04:37.408522 5081 heartbeater.cc:510] Master 127.1.109.126:34547 requested a full tablet report, sending...
I20250114 03:04:37.409545 4628 ts_manager.cc:194] Registered new tserver with Master: 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:37.410105 4628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:60231
I20250114 03:04:37.417758 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:04:37.419155 1461 test_util.cc:274] Using random seed: -767541106
I20250114 03:04:37.428813 4628 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53890:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 03:04:37.429975 4628 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:04:37.446337 5014 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae6c5627ba43f1b3a5ecf476539b81 (DEFAULT_TABLE table=test-workload [id=fe2361ee439444678dbcfcd3945548ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:04:37.446345 4751 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae6c5627ba43f1b3a5ecf476539b81 (DEFAULT_TABLE table=test-workload [id=fe2361ee439444678dbcfcd3945548ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:04:37.446344 4882 tablet_service.cc:1467] Processing CreateTablet for tablet 82ae6c5627ba43f1b3a5ecf476539b81 (DEFAULT_TABLE table=test-workload [id=fe2361ee439444678dbcfcd3945548ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:04:37.447350 5014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ae6c5627ba43f1b3a5ecf476539b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:37.447348 4882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ae6c5627ba43f1b3a5ecf476539b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:37.447348 4751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82ae6c5627ba43f1b3a5ecf476539b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:04:37.452661 5098 tablet_bootstrap.cc:492] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746: Bootstrap starting.
I20250114 03:04:37.454223 5099 tablet_bootstrap.cc:492] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32: Bootstrap starting.
I20250114 03:04:37.454316 5100 tablet_bootstrap.cc:492] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd: Bootstrap starting.
I20250114 03:04:37.455142 5098 tablet_bootstrap.cc:654] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:37.456038 5098 log.cc:826] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:37.456207 5099 tablet_bootstrap.cc:654] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:37.456511 5100 tablet_bootstrap.cc:654] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd: Neither blocks nor log segments found. Creating new log.
I20250114 03:04:37.456972 5099 log.cc:826] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:37.457270 5100 log.cc:826] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd: Log is configured to *not* fsync() on all Append() calls
I20250114 03:04:37.457998 5098 tablet_bootstrap.cc:492] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746: No bootstrap required, opened a new log
I20250114 03:04:37.458186 5098 ts_tablet_manager.cc:1397] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250114 03:04:37.458595 5099 tablet_bootstrap.cc:492] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32: No bootstrap required, opened a new log
I20250114 03:04:37.458773 5099 ts_tablet_manager.cc:1397] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:04:37.460258 5100 tablet_bootstrap.cc:492] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd: No bootstrap required, opened a new log
I20250114 03:04:37.460438 5100 ts_tablet_manager.cc:1397] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250114 03:04:37.461833 5099 raft_consensus.cc:357] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.462036 5099 raft_consensus.cc:383] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:37.462113 5099 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f5a4e02dd1463db75006734f788a32, State: Initialized, Role: FOLLOWER
I20250114 03:04:37.462127 5098 raft_consensus.cc:357] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.462371 5098 raft_consensus.cc:383] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:37.462461 5098 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 879969dde1ff460bb29cf9319dcd8746, State: Initialized, Role: FOLLOWER
I20250114 03:04:37.462566 5099 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.462909 5098 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.463599 5099 ts_tablet_manager.cc:1428] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:04:37.463874 5098 ts_tablet_manager.cc:1428] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 03:04:37.464056 4817 heartbeater.cc:502] Master 127.1.109.126:34547 was elected leader, sending a full tablet report...
I20250114 03:04:37.464268 5081 heartbeater.cc:502] Master 127.1.109.126:34547 was elected leader, sending a full tablet report...
I20250114 03:04:37.464587 5106 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.464717 5106 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.465106 5100 raft_consensus.cc:357] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.465317 5100 raft_consensus.cc:383] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:04:37.465384 5100 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77a32f4dd261435cb6dc91526239a9cd, State: Initialized, Role: FOLLOWER
I20250114 03:04:37.465433 5107 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.465575 5107 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.465860 5100 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.466557 5100 ts_tablet_manager.cc:1428] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 03:04:37.466908 4948 heartbeater.cc:502] Master 127.1.109.126:34547 was elected leader, sending a full tablet report...
I20250114 03:04:37.467376 5107 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.467758 5108 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.468115 5108 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.470872 5108 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:37.473305 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
W20250114 03:04:37.473488 4968 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.65:37273 timed out after 0.006s (SENT)
I20250114 03:04:37.473541 4771 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 0.
I20250114 03:04:37.475358 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.475521 4771 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a32f4dd261435cb6dc91526239a9cd in term 0.
I20250114 03:04:37.475853 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.475987 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32; no voters:
I20250114 03:04:37.476054 5034 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a32f4dd261435cb6dc91526239a9cd in term 0.
I20250114 03:04:37.476293 5108 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:04:37.476361 5108 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:37.476989 5106 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.476404 5108 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:37.478873 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.479032 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.479151 4771 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 0.
I20250114 03:04:37.479740 4968 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32; no voters:
I20250114 03:04:37.480320 5107 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:04:37.480399 5107 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:37.480450 5107 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:37.480731 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.480875 5108 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.481285 5108 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 1 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:37.481776 4902 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 879969dde1ff460bb29cf9319dcd8746 in current term 1: Already voted for candidate 77a32f4dd261435cb6dc91526239a9cd in this term.
I20250114 03:04:37.481755 5107 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.481819 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746"
I20250114 03:04:37.482187 5107 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 1 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
W20250114 03:04:37.482618 4704 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.1.109.67:44225 timed out after 0.006s (SENT)
I20250114 03:04:37.482932 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:37.483088 4902 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 879969dde1ff460bb29cf9319dcd8746 in current term 1: Already voted for candidate 77a32f4dd261435cb6dc91526239a9cd in this term.
I20250114 03:04:37.483340 5034 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 in current term 1: Already voted for candidate 879969dde1ff460bb29cf9319dcd8746 in this term.
I20250114 03:04:37.484018 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:37.484143 4770 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:04:37.484304 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:37.484875 4968 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746; no voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.485123 5107 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250114 03:04:37.485352 4770 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a32f4dd261435cb6dc91526239a9cd in term 1.
I20250114 03:04:37.485668 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32; no voters: 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:37.485775 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.485900 5034 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 in current term 1: Already voted for candidate 879969dde1ff460bb29cf9319dcd8746 in this term.
I20250114 03:04:37.485961 5108 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:04:37.486064 5108 raft_consensus.cc:695] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 LEADER]: Becoming Leader. State: Replica: 77a32f4dd261435cb6dc91526239a9cd, State: Running, Role: LEADER
I20250114 03:04:37.486415 5108 consensus_queue.cc:237] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.488232 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.488652 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:37.489079 5106 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250114 03:04:37.490167 4629 catalog_manager.cc:5526] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd reported cstate change: term changed from 0 to 1, leader changed from <none> to 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66). New cstate: current_term: 1 leader_uuid: "77a32f4dd261435cb6dc91526239a9cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } health_report { overall_health: HEALTHY } } }
I20250114 03:04:37.490842 5109 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:04:37.491117 5108 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:04:37.493116 5107 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77a32f4dd261435cb6dc91526239a9cd)
I20250114 03:04:37.493225 5107 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.493608 5107 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.493958 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.493914 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.494073 4771 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 1.
I20250114 03:04:37.494365 4968 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32; no voters:
I20250114 03:04:37.495425 5107 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 03:04:37.495481 5107 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 77a32f4dd261435cb6dc91526239a9cd)
I20250114 03:04:37.495535 5107 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:04:37.496505 5107 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.496917 5107 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 2 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.497200 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:37.497845 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:37.498212 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746; no voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.498662 5107 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 03:04:37.498919 5034 raft_consensus.cc:1235] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Rejecting Update request from peer 77a32f4dd261435cb6dc91526239a9cd for earlier term 1. Current term is 2. Ops: []
I20250114 03:04:37.499315 5109 consensus_queue.cc:1046] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:04:37.499847 5121 raft_consensus.cc:3049] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:04:37.499912 5121 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 77a32f4dd261435cb6dc91526239a9cd, State: Running, Role: LEADER
I20250114 03:04:37.500095 5121 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.500340 5121 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:04:37.504269 5121 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.504379 5121 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.504786 5121 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:37.505125 5105 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.505124 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.505242 5105 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.505434 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.505561 5034 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a32f4dd261435cb6dc91526239a9cd in term 2.
I20250114 03:04:37.505678 5105 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.506006 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.506103 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters: f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.506131 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.506264 4902 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 2.
I20250114 03:04:37.506337 5121 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 03:04:37.506394 5121 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:37.506433 5121 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:04:37.506758 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters: f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.507004 5105 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 03:04:37.507066 5105 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:37.507103 5105 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:04:37.507100 5106 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 77a32f4dd261435cb6dc91526239a9cd)
I20250114 03:04:37.507189 5106 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.507347 5121 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.507512 5106 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.507946 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.508080 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746"
I20250114 03:04:37.508332 5105 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.508481 5121 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 3 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:37.508538 5034 raft_consensus.cc:2363] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 for earlier term 2. Current term is 3.
I20250114 03:04:37.508695 5105 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 3 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.509069 4704 leader_election.cc:400] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225) with higher term. Message: Invalid argument: T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 for earlier term 2. Current term is 3.
I20250114 03:04:37.509115 4704 leader_election.cc:403] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20250114 03:04:37.509120 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:37.509266 4770 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 1 FOLLOWER]: Advancing to term 3
I20250114 03:04:37.509259 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:37.509320 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:37.509409 4901 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 879969dde1ff460bb29cf9319dcd8746 in current term 3: Already voted for candidate 77a32f4dd261435cb6dc91526239a9cd in this term.
I20250114 03:04:37.509421 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.509547 4902 raft_consensus.cc:2363] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 for earlier term 2. Current term is 3.
I20250114 03:04:37.509814 4968 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746; no voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.509858 4705 leader_election.cc:400] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591) with higher term. Message: Invalid argument: T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 for earlier term 2. Current term is 3.
I20250114 03:04:37.510023 5105 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20250114 03:04:37.510270 4770 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77a32f4dd261435cb6dc91526239a9cd in term 3.
I20250114 03:04:37.510411 5106 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225) with higher term. Message: Invalid argument: T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 for earlier term 2. Current term is 3.
I20250114 03:04:37.510564 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32; no voters: 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:37.510726 5121 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:04:37.510826 5121 raft_consensus.cc:695] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 LEADER]: Becoming Leader. State: Replica: 77a32f4dd261435cb6dc91526239a9cd, State: Running, Role: LEADER
I20250114 03:04:37.510967 5121 consensus_queue.cc:237] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.512972 4628 catalog_manager.cc:5526] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "77a32f4dd261435cb6dc91526239a9cd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } health_report { overall_health: HEALTHY } } }
I20250114 03:04:37.513568 5034 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Refusing update from remote peer 77a32f4dd261435cb6dc91526239a9cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 03:04:37.514614 4770 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Refusing update from remote peer 77a32f4dd261435cb6dc91526239a9cd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 03:04:37.515954 5121 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:04:37.516171 5108 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:04:37.516770 5106 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.516901 5106 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.517287 5106 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.518103 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.518191 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.518316 5034 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f5a4e02dd1463db75006734f788a32 in term 3.
I20250114 03:04:37.518898 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd
I20250114 03:04:37.519183 5104 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 03:04:37.519248 5104 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 77a32f4dd261435cb6dc91526239a9cd)
I20250114 03:04:37.519304 5104 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:04:37.520392 5104 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.520690 5104 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 4 election: Requested vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
W20250114 03:04:37.520933 5119 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:04:37.520978 5119 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250114 03:04:37.521147 4770 raft_consensus.cc:1235] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Rejecting Update request from peer 77a32f4dd261435cb6dc91526239a9cd for earlier term 3. Current term is 4. Ops: []
I20250114 03:04:37.521188 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:37.521610 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746"
I20250114 03:04:37.521606 5109 consensus_queue.cc:1046] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:04:37.521770 5109 raft_consensus.cc:3049] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 LEADER]: Stepping down as leader of term 3
I20250114 03:04:37.521814 5109 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 77a32f4dd261435cb6dc91526239a9cd, State: Running, Role: LEADER
I20250114 03:04:37.521935 5109 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.521994 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:37.522081 5109 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:04:37.522208 5104 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20250114 03:04:37.523624 4862 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 77a32f4dd261435cb6dc91526239a9cd is not leader of this config: current role FOLLOWER
I20250114 03:04:37.528347 5105 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 77a32f4dd261435cb6dc91526239a9cd)
I20250114 03:04:37.528451 5105 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.528867 5121 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.528972 5121 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.529058 5105 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.529382 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.529433 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.529560 4902 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 4.
I20250114 03:04:37.529603 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.529732 4770 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 879969dde1ff460bb29cf9319dcd8746 in current term 4: Already voted for candidate f6f5a4e02dd1463db75006734f788a32 in this term.
I20250114 03:04:37.530126 5104 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.530212 5104 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.530267 5121 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:37.530546 5104 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.530881 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.530881 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.531002 5034 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f5a4e02dd1463db75006734f788a32 in term 3.
I20250114 03:04:37.531002 4902 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6f5a4e02dd1463db75006734f788a32 in term 4.
I20250114 03:04:37.531251 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32; no voters:
I20250114 03:04:37.531397 5104 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250114 03:04:37.531441 5104 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:37.531467 5104 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 4 FOLLOWER]: Advancing to term 5
W20250114 03:04:37.532285 4731 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45176: Illegal state: replica f6f5a4e02dd1463db75006734f788a32 is not leader of this config: current role FOLLOWER
I20250114 03:04:37.532299 5104 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.532678 5104 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 5 election: Requested vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.533051 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746"
I20250114 03:04:37.533179 5034 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 3 FOLLOWER]: Advancing to term 5
I20250114 03:04:37.533262 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:37.533272 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters: f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.533372 4902 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 4 FOLLOWER]: Advancing to term 5
I20250114 03:04:37.533540 5033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.533907 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.534293 5034 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f5a4e02dd1463db75006734f788a32 in term 5.
I20250114 03:04:37.534389 4902 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6f5a4e02dd1463db75006734f788a32 in term 5.
I20250114 03:04:37.534508 5105 raft_consensus.cc:2758] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20250114 03:04:37.534655 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32; no voters:
I20250114 03:04:37.534795 5121 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20250114 03:04:37.534970 5104 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 5 FOLLOWER]: Leader election won for term 5
I20250114 03:04:37.535086 5104 raft_consensus.cc:695] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 5 LEADER]: Becoming Leader. State: Replica: f6f5a4e02dd1463db75006734f788a32, State: Running, Role: LEADER
I20250114 03:04:37.535372 5104 consensus_queue.cc:237] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.537453 4628 catalog_manager.cc:5526] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 reported cstate change: term changed from 3 to 5, leader changed from 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66) to f6f5a4e02dd1463db75006734f788a32 (127.1.109.65). New cstate: current_term: 5 leader_uuid: "f6f5a4e02dd1463db75006734f788a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } health_report { overall_health: UNKNOWN } } }
I20250114 03:04:37.537690 5034 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Refusing update from remote peer f6f5a4e02dd1463db75006734f788a32: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 03:04:37.538165 5104 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:04:37.538688 4902 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 5 FOLLOWER]: Refusing update from remote peer f6f5a4e02dd1463db75006734f788a32: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20250114 03:04:37.539206 5106 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20250114 03:04:37.545231 4994 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38322: Illegal state: replica 879969dde1ff460bb29cf9319dcd8746 is not leader of this config: current role FOLLOWER
W20250114 03:04:37.556710 5122 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:04:37.556787 5122 fault_injection.cc:44] THIS SERVER MAY CRASH!
W20250114 03:04:37.565650 5119 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: Replica 77a32f4dd261435cb6dc91526239a9cd is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "f6f5a4e02dd1463db75006734f788a32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } } }
I20250114 03:04:37.712328 5105 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f6f5a4e02dd1463db75006734f788a32)
I20250114 03:04:37.712482 5105 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.712962 5105 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.713536 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:37.713577 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.713724 4902 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 5.
I20250114 03:04:37.714032 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters:
I20250114 03:04:37.714215 5105 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250114 03:04:37.714262 5105 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f6f5a4e02dd1463db75006734f788a32)
I20250114 03:04:37.714306 5105 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:04:37.715461 5105 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.715880 5105 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 6 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.716574 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:37.716706 4901 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:04:37.717908 4901 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 6.
I20250114 03:04:37.718101 4902 raft_consensus.cc:1235] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Rejecting Update request from peer f6f5a4e02dd1463db75006734f788a32 for earlier term 5. Current term is 6. Ops: []
I20250114 03:04:37.718152 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:37.718503 5034 raft_consensus.cc:1235] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 FOLLOWER]: Rejecting Update request from peer f6f5a4e02dd1463db75006734f788a32 for earlier term 5. Current term is 6. Ops: []
I20250114 03:04:37.718912 5127 consensus_queue.cc:1046] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 03:04:37.718981 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters: f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:37.719134 5105 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 FOLLOWER]: Leader election won for term 6
I20250114 03:04:37.719172 5127 raft_consensus.cc:3049] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 5 LEADER]: Stepping down as leader of term 5
I20250114 03:04:37.719230 5127 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f6f5a4e02dd1463db75006734f788a32, State: Running, Role: LEADER
I20250114 03:04:37.719241 5105 raft_consensus.cc:695] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 LEADER]: Becoming Leader. State: Replica: 879969dde1ff460bb29cf9319dcd8746, State: Running, Role: LEADER
I20250114 03:04:37.719396 5127 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.719544 5105 consensus_queue.cc:237] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.720525 5106 consensus_queue.cc:1046] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 03:04:37.721292 5127 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 5 FOLLOWER]: Advancing to term 6
I20250114 03:04:37.721804 4627 catalog_manager.cc:5526] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 reported cstate change: term changed from 5 to 6, leader changed from f6f5a4e02dd1463db75006734f788a32 (127.1.109.65) to 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67). New cstate: current_term: 6 leader_uuid: "879969dde1ff460bb29cf9319dcd8746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } health_report { overall_health: UNKNOWN } } }
I20250114 03:04:37.722090 4902 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Refusing update from remote peer 879969dde1ff460bb29cf9319dcd8746: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 03:04:37.722478 5105 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 03:04:37.723034 4770 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Refusing update from remote peer 879969dde1ff460bb29cf9319dcd8746: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20250114 03:04:37.723506 5105 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20250114 03:04:37.723824 5104 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:37.723912 5104 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:37.724246 5104 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:37.725706 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:37.727541 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:37.728082 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:37.728471 5127 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:38.220456 5127 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:38.220615 5127 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:38.221138 5127 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:38.221565 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:38.221619 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:38.222141 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:38.222309 5127 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250114 03:04:38.473534 5122 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45176: Illegal state: Replica f6f5a4e02dd1463db75006734f788a32 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 leader_uuid: "879969dde1ff460bb29cf9319dcd8746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } } }
W20250114 03:04:38.474298 5122 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45176: Illegal state: Cannot assign timestamp to op. Tablet is not in leader mode. Last heard from a leader: 0.001s ago.
W20250114 03:04:38.482770 5129 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:04:38.482836 5129 fault_injection.cc:44] THIS SERVER MAY CRASH!
I20250114 03:04:39.419589 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:39.419732 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:39.420418 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:39.420922 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:39.420928 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:39.421402 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:39.421622 5132 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:39.886559 5131 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:39.886704 5131 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:39.887341 5131 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:39.887702 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:39.887805 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:39.888300 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:40.323104 5131 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:40.325003 5141 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=1, attempt_no=2}) took 1843 ms. Trace:
I20250114 03:04:40.325120 5141 rpcz_store.cc:276] 0114 03:04:38.481946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:38.481981 (+ 35us) service_pool.cc:225] Handling call
0114 03:04:40.324984 (+1843003us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:39.027798 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:39.027912 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:39.027917 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:39.027919 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:39.027983 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:39.028008 (+ 25us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:39.028010 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:39.028046 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:04:39.028046 (+ 0us) write_op.cc:260] PREPARE: finished
0114 03:04:39.028084 (+ 38us) write_op.cc:270] Start()
0114 03:04:39.699212 (+671128us) write_op.cc:276] Timestamp: P: 1736823879028049 usec, L: 0
0114 03:04:39.699215 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:39.699371 (+ 156us) log.cc:844] Serialized 345 byte log entry
0114 03:04:40.324322 (+624951us) op_driver.cc:464] REPLICATION: finished
0114 03:04:40.324543 (+ 221us) write_op.cc:301] APPLY: starting
0114 03:04:40.324562 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:40.324592 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:40.324608 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:40.324674 (+ 66us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:40.324703 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:40.324706 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:40.324790 (+ 84us) log.cc:844] Serialized 22 byte log entry
0114 03:04:40.324891 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:40.324903 (+ 12us) write_op.cc:454] Released schema lock
0114 03:04:40.324908 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":156,"num_ops":1,"prepare.queue_time_us":545540,"prepare.run_cpu_time_us":566,"prepare.run_wall_time_us":671633,"replication_time_us":625074,"thread_start_us":91,"threads_started":1}]]}
I20250114 03:04:40.358424 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:40.358562 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:40.359124 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:40.359557 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:40.359560 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:40.359993 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:40.752877 5131 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:40.753078 5131 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:40.753525 5131 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:40.754011 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:40.754175 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:40.754246 5141 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=2, attempt_no=0}) took 1330 ms. Trace:
I20250114 03:04:40.754329 5141 rpcz_store.cc:276] 0114 03:04:39.423916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:39.423980 (+ 64us) service_pool.cc:225] Handling call
0114 03:04:40.754232 (+1330252us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:39.699464 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:39.699575 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:39.699580 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:39.699581 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:39.699659 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:39.699668 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:39.699674 (+ 6us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:39.699722 (+ 48us) tablet.cc:698] Row locks acquired
0114 03:04:39.699723 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:39.699764 (+ 41us) write_op.cc:270] Start()
0114 03:04:39.997942 (+298178us) write_op.cc:276] Timestamp: P: 1736823879699722 usec, L: 0
0114 03:04:39.997947 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:39.998170 (+ 223us) log.cc:844] Serialized 354 byte log entry
0114 03:04:40.753723 (+755553us) op_driver.cc:464] REPLICATION: finished
0114 03:04:40.753835 (+ 112us) write_op.cc:301] APPLY: starting
0114 03:04:40.753858 (+ 23us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:40.753890 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:40.753910 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:40.753971 (+ 61us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:40.754011 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:40.754015 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:40.754085 (+ 70us) log.cc:844] Serialized 22 byte log entry
0114 03:04:40.754142 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:40.754155 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:40.754160 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":46,"num_ops":1,"prepare.queue_time_us":275152,"prepare.run_cpu_time_us":695,"prepare.run_wall_time_us":298775,"replication_time_us":755738}]]}
I20250114 03:04:40.754662 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:41.245060 5132 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:41.526314 5131 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:41.527776 5148 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=3, attempt_no=0}) took 1200 ms. Trace:
I20250114 03:04:41.527850 5148 rpcz_store.cc:276] 0114 03:04:40.327517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:40.327580 (+ 63us) service_pool.cc:225] Handling call
0114 03:04:41.527762 (+1200182us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:40.328026 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:40.328118 (+ 92us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:40.328121 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:40.328122 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:40.328184 (+ 62us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:40.328190 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:40.328193 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:40.328224 (+ 31us) tablet.cc:698] Row locks acquired
0114 03:04:40.328224 (+ 0us) write_op.cc:260] PREPARE: finished
0114 03:04:40.328250 (+ 26us) write_op.cc:270] Start()
0114 03:04:40.711468 (+383218us) write_op.cc:276] Timestamp: P: 1736823880328201 usec, L: 0
0114 03:04:40.711473 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:40.711692 (+ 219us) log.cc:844] Serialized 354 byte log entry
0114 03:04:41.527235 (+815543us) op_driver.cc:464] REPLICATION: finished
0114 03:04:41.527399 (+ 164us) write_op.cc:301] APPLY: starting
0114 03:04:41.527415 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:41.527442 (+ 27us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:41.527461 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:41.527510 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:41.527533 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:41.527536 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:41.527607 (+ 71us) log.cc:844] Serialized 22 byte log entry
0114 03:04:41.527668 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:41.527685 (+ 17us) write_op.cc:454] Released schema lock
0114 03:04:41.527689 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":117,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":383744,"replication_time_us":815731,"thread_start_us":77,"threads_started":1}]]}
I20250114 03:04:41.915710 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:41.915860 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:41.916347 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:41.916783 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:41.916801 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:41.917263 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:42.094931 5131 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:42.095103 5131 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:42.095638 5131 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:42.096123 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:42.096375 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 9 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:42.096843 5154 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=4, attempt_no=0}) took 1341 ms. Trace:
I20250114 03:04:42.096858 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:42.096920 5154 rpcz_store.cc:276] 0114 03:04:40.755727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:40.755786 (+ 59us) service_pool.cc:225] Handling call
0114 03:04:42.096801 (+1341015us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:40.756199 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:40.756283 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:40.756286 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:40.756288 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:40.756340 (+ 52us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:40.756347 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:40.756349 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:40.756390 (+ 41us) tablet.cc:698] Row locks acquired
0114 03:04:40.756392 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:40.756417 (+ 25us) write_op.cc:270] Start()
0114 03:04:41.499567 (+743150us) write_op.cc:276] Timestamp: P: 1736823880756373 usec, L: 0
0114 03:04:41.499571 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:41.499786 (+ 215us) log.cc:844] Serialized 354 byte log entry
0114 03:04:42.095948 (+596162us) op_driver.cc:464] REPLICATION: finished
0114 03:04:42.096229 (+ 281us) write_op.cc:301] APPLY: starting
0114 03:04:42.096249 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:42.096282 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:42.096311 (+ 29us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:42.096366 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:42.096393 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:42.096397 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:42.096460 (+ 63us) log.cc:844] Serialized 22 byte log entry
0114 03:04:42.096696 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:42.096711 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:42.096716 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":229,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":581,"prepare.run_wall_time_us":743645,"replication_time_us":596344,"thread_start_us":108,"threads_started":1}]]}
I20250114 03:04:42.452847 5132 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:42.622525 5131 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:43.015610 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:43.015749 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.016505 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:43.016961 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:43.016968 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:43.017506 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:43.017747 5132 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:43.440538 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:43.440729 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.440964 5131 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:43.441068 5131 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.441373 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:43.441536 5131 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:43.441895 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:43.442517 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:43.442545 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:43.442663 4770 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77a32f4dd261435cb6dc91526239a9cd in term 6.
I20250114 03:04:43.443256 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, f6f5a4e02dd1463db75006734f788a32; no voters: 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:43.443500 5132 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20250114 03:04:43.443563 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Starting leader election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:43.443622 5132 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:04:43.443934 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 7 candidate_status { last_received { term: 6 index: 11 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:43.444900 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.445158 4901 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f6f5a4e02dd1463db75006734f788a32 in current term 7: Already voted for candidate 77a32f4dd261435cb6dc91526239a9cd in this term.
I20250114 03:04:43.445341 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:43.445396 4902 raft_consensus.cc:1235] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Rejecting Update request from peer 879969dde1ff460bb29cf9319dcd8746 for earlier term 6. Current term is 7. Ops: []
I20250114 03:04:43.445665 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:43.445974 5165 consensus_queue.cc:1046] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 }, Status: INVALID_TERM, Last received: 6.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20250114 03:04:43.446091 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:43.446080 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 7 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746"
I20250114 03:04:43.446198 5147 raft_consensus.cc:3049] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 LEADER]: Stepping down as leader of term 6
I20250114 03:04:43.446257 5147 raft_consensus.cc:738] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 879969dde1ff460bb29cf9319dcd8746, State: Running, Role: LEADER
I20250114 03:04:43.446439 5147 consensus_queue.cc:260] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 6.12, Last appended by leader: 12, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.446547 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:43.446717 5147 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:04:43.446730 5132 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20250114 03:04:43.452934 5147 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:43.453069 5147 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.453675 5147 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:43.454119 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 8 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:43.454140 5132 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:04:43.454129 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 8 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:43.454267 5132 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.454448 4901 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 7.
I20250114 03:04:43.454775 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters: f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:43.454968 5147 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20250114 03:04:43.455034 5147 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:04:43.455088 5147 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:04:43.455061 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:43.455191 5132 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:43.455389 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 8 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:43.456113 5147 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.456326 5034 raft_consensus.cc:2388] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 77a32f4dd261435cb6dc91526239a9cd in current term 8: Already voted for candidate 879969dde1ff460bb29cf9319dcd8746 in this term.
I20250114 03:04:43.456460 5147 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 8 election: Requested vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:43.456678 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:43.456784 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 8 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32"
I20250114 03:04:43.456876 5132 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:04:43.456923 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "879969dde1ff460bb29cf9319dcd8746" candidate_term: 8 candidate_status { last_received { term: 6 index: 12 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd"
I20250114 03:04:43.457731 5132 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20250114 03:04:43.458531 4901 raft_consensus.cc:2463] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 879969dde1ff460bb29cf9319dcd8746 in term 8.
I20250114 03:04:43.458812 4967 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746; no voters: f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:43.458951 5147 raft_consensus.cc:2798] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 8 FOLLOWER]: Leader election won for term 8
I20250114 03:04:43.459029 5147 raft_consensus.cc:695] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [term 8 LEADER]: Becoming Leader. State: Replica: 879969dde1ff460bb29cf9319dcd8746, State: Running, Role: LEADER
I20250114 03:04:43.459177 5147 consensus_queue.cc:237] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 6.12, Last appended by leader: 12, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:43.460855 4628 catalog_manager.cc:5526] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "879969dde1ff460bb29cf9319dcd8746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } health_report { overall_health: UNKNOWN } } }
I20250114 03:04:43.461719 4901 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Refusing update from remote peer 879969dde1ff460bb29cf9319dcd8746: Log matching property violated. Preceding OpId in replica: term: 6 index: 12. Preceding OpId from leader: term: 8 index: 13. (index mismatch)
I20250114 03:04:43.462111 5165 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20250114 03:04:44.027128 5129 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38322: Aborted: Op submitted in term 6 cannot be replicated in term 8
I20250114 03:04:44.027328 5129 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=8, attempt_no=0}) took 1008 ms. Trace:
I20250114 03:04:44.027384 5129 rpcz_store.cc:276] 0114 03:04:43.019076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:43.019130 (+ 54us) service_pool.cc:225] Handling call
0114 03:04:44.027316 (+1008186us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:43.428583 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:43.428699 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:43.428703 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:43.428704 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:43.428775 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:43.428783 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:43.428786 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:43.428851 (+ 65us) tablet.cc:698] Row locks acquired
0114 03:04:43.428854 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:04:43.428900 (+ 46us) write_op.cc:270] Start()
0114 03:04:44.027040 (+598140us) write_op.cc:276] Timestamp: P: 1736823883428844 usec, L: 0
0114 03:04:44.027044 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:44.027080 (+ 36us) op_driver.cc:389] HandleFailure(Aborted: Op submitted in term 6 cannot be replicated in term 8)
0114 03:04:44.027097 (+ 17us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:44.027110 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:44.027115 (+ 5us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":409092}]]}
W20250114 03:04:44.033053 4731 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45176: Illegal state: replica f6f5a4e02dd1463db75006734f788a32 is not leader of this config: current role FOLLOWER
W20250114 03:04:44.036494 4862 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 77a32f4dd261435cb6dc91526239a9cd is not leader of this config: current role FOLLOWER
W20250114 03:04:44.113322 5129 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38322: Aborted: Op submitted in term 6 cannot be replicated in term 8
W20250114 03:04:44.118901 4731 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45176: Illegal state: replica f6f5a4e02dd1463db75006734f788a32 is not leader of this config: current role FOLLOWER
W20250114 03:04:44.126399 4862 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:54438: Illegal state: replica 77a32f4dd261435cb6dc91526239a9cd is not leader of this config: current role FOLLOWER
I20250114 03:04:44.308544 5131 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 6 FOLLOWER]: Advancing to term 7
I20250114 03:04:44.309787 5131 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250114 03:04:44.310065 4771 raft_consensus.cc:3054] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 7 FOLLOWER]: Advancing to term 8
I20250114 03:04:44.310832 4771 raft_consensus.cc:1270] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Refusing update from remote peer 879969dde1ff460bb29cf9319dcd8746: Log matching property violated. Preceding OpId in replica: term: 6 index: 12. Preceding OpId from leader: term: 8 index: 13. (index mismatch)
I20250114 03:04:44.311506 5147 consensus_queue.cc:1035] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250114 03:04:45.428396 5176 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:45.428555 5176 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:45.429095 5176 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:45.429540 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 14 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:45.429574 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 14 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:45.430076 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:45.430253 5176 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:45.430538 5180 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=8, attempt_no=3}) took 1386 ms. Trace:
I20250114 03:04:45.430619 5180 rpcz_store.cc:276] 0114 03:04:44.043796 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:44.043860 (+ 64us) service_pool.cc:225] Handling call
0114 03:04:45.430513 (+1386653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:44.113761 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:44.113871 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:44.113876 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:44.113878 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:44.113953 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:44.113961 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:44.113964 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:44.114022 (+ 58us) tablet.cc:698] Row locks acquired
0114 03:04:44.114023 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:44.114054 (+ 31us) write_op.cc:270] Start()
0114 03:04:45.093193 (+979139us) write_op.cc:276] Timestamp: P: 1736823884114021 usec, L: 0
0114 03:04:45.093196 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:45.093378 (+ 182us) log.cc:844] Serialized 354 byte log entry
0114 03:04:45.429679 (+336301us) op_driver.cc:464] REPLICATION: finished
0114 03:04:45.430119 (+ 440us) write_op.cc:301] APPLY: starting
0114 03:04:45.430135 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:45.430169 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:45.430185 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:45.430242 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:45.430270 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:45.430274 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:45.430335 (+ 61us) log.cc:844] Serialized 22 byte log entry
0114 03:04:45.430416 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:45.430431 (+ 15us) write_op.cc:454] Released schema lock
0114 03:04:45.430435 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":391,"num_ops":1,"prepare.queue_time_us":69508,"prepare.run_cpu_time_us":733,"prepare.run_wall_time_us":979791,"replication_time_us":336452,"spinlock_wait_cycles":40192,"thread_start_us":185,"threads_started":2,"wal-append.queue_time_us":168}]]}
I20250114 03:04:45.793318 5177 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:45.793452 5177 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:45.793962 5177 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:45.794267 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 14 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:45.794565 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 14 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:45.795063 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:45.795248 5177 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:46.598948 5176 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:46.599102 5176 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:46.599623 5176 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:46.600018 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:46.600057 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:46.600481 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:46.600728 5176 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:46.600904 5189 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=9, attempt_no=3}) took 2465 ms. Trace:
I20250114 03:04:46.600973 5189 rpcz_store.cc:276] 0114 03:04:44.135582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:44.135650 (+ 68us) service_pool.cc:225] Handling call
0114 03:04:46.600874 (+2465224us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:45.093588 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:45.093694 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:45.093699 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:45.093700 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:45.093798 (+ 98us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:45.093809 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:45.093813 (+ 4us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:45.093858 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:04:45.093859 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:45.093900 (+ 41us) write_op.cc:270] Start()
0114 03:04:45.864047 (+770147us) write_op.cc:276] Timestamp: P: 1736823885093867 usec, L: 0
0114 03:04:45.864052 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:45.864240 (+ 188us) log.cc:844] Serialized 354 byte log entry
0114 03:04:46.600179 (+735939us) op_driver.cc:464] REPLICATION: finished
0114 03:04:46.600425 (+ 246us) write_op.cc:301] APPLY: starting
0114 03:04:46.600445 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:46.600480 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:46.600495 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:46.600557 (+ 62us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:46.600589 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:46.600594 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:46.600665 (+ 71us) log.cc:844] Serialized 22 byte log entry
0114 03:04:46.600749 (+ 84us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:46.600763 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:46.600768 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"num_ops":1,"prepare.queue_time_us":957522,"prepare.run_cpu_time_us":648,"prepare.run_wall_time_us":770719,"replication_time_us":736096,"thread_start_us":80,"threads_started":1}]]}
I20250114 03:04:46.762176 5177 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:46.762332 5177 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:46.762861 5177 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:46.763226 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:46.763295 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 15 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:46.763773 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:47.073984 5176 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:47.074146 5176 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:47.074761 5176 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:47.075138 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 16 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:47.075253 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 16 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:47.075714 5189 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=10, attempt_no=0}) took 1643 ms. Trace:
I20250114 03:04:47.075731 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:47.075872 5189 rpcz_store.cc:276] 0114 03:04:45.431969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:45.432027 (+ 58us) service_pool.cc:225] Handling call
0114 03:04:47.075701 (+1643674us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:45.864349 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:45.864464 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:45.864469 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:45.864471 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:45.864548 (+ 77us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:45.864559 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:45.864562 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:45.864608 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:04:45.864612 (+ 4us) write_op.cc:260] PREPARE: finished
0114 03:04:45.864656 (+ 44us) write_op.cc:270] Start()
0114 03:04:46.554840 (+690184us) write_op.cc:276] Timestamp: P: 1736823885864601 usec, L: 0
0114 03:04:46.554844 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:46.555072 (+ 228us) log.cc:844] Serialized 354 byte log entry
0114 03:04:47.075071 (+519999us) op_driver.cc:464] REPLICATION: finished
0114 03:04:47.075179 (+ 108us) write_op.cc:301] APPLY: starting
0114 03:04:47.075196 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:47.075228 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:47.075244 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:47.075293 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:47.075336 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:47.075342 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:04:47.075541 (+ 199us) log.cc:844] Serialized 43 byte log entry
0114 03:04:47.075595 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:47.075607 (+ 12us) write_op.cc:454] Released schema lock
0114 03:04:47.075612 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"num_ops":1,"prepare.queue_time_us":431961,"prepare.run_cpu_time_us":707,"prepare.run_wall_time_us":690786,"replication_time_us":520191}]]}
I20250114 03:04:47.235486 5177 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:47.801165 5177 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:47.801306 5177 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:47.801883 5177 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:47.802299 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:47.802294 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 18 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:47.802726 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:47.803216 5195 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=11, attempt_no=0}) took 1201 ms. Trace:
I20250114 03:04:47.803416 5177 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:47.803332 5195 rpcz_store.cc:276] 0114 03:04:46.602163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:46.602222 (+ 59us) service_pool.cc:225] Handling call
0114 03:04:47.803202 (+1200980us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:46.602620 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:46.602719 (+ 99us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:46.602723 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:46.602724 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:46.602777 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:46.602783 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:46.602785 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:46.602816 (+ 31us) tablet.cc:698] Row locks acquired
0114 03:04:46.602817 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:46.602844 (+ 27us) write_op.cc:270] Start()
0114 03:04:47.021004 (+418160us) write_op.cc:276] Timestamp: P: 1736823886602824 usec, L: 0
0114 03:04:47.021008 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:47.021242 (+ 234us) log.cc:844] Serialized 354 byte log entry
0114 03:04:47.802596 (+781354us) op_driver.cc:464] REPLICATION: finished
0114 03:04:47.802818 (+ 222us) write_op.cc:301] APPLY: starting
0114 03:04:47.802836 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:47.802866 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:47.802880 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:47.802932 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:47.802971 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:47.802974 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:47.803043 (+ 69us) log.cc:844] Serialized 43 byte log entry
0114 03:04:47.803097 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:47.803109 (+ 12us) write_op.cc:454] Released schema lock
0114 03:04:47.803115 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":165,"num_ops":1,"prepare.queue_time_us":49,"prepare.run_cpu_time_us":627,"prepare.run_wall_time_us":418690,"replication_time_us":781557,"thread_start_us":56,"threads_started":1}]]}
I20250114 03:04:47.877583 5176 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:48.609290 5177 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:48.609423 5177 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:48.609949 5177 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:48.610349 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:48.610407 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 19 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:48.610810 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:48.611068 5177 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:49.560344 5177 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:49.560487 5177 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:49.561141 5177 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:49.561527 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:49.561529 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:49.562027 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:49.562218 5177 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:49.562448 5212 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=14, attempt_no=0}) took 1757 ms. Trace:
I20250114 03:04:49.562510 5212 rpcz_store.cc:276] 0114 03:04:47.805228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:47.805280 (+ 52us) service_pool.cc:225] Handling call
0114 03:04:49.562432 (+1757152us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:47.868936 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:47.869056 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:47.869065 (+ 9us) write_op.cc:435] Acquired schema lock
0114 03:04:47.869066 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:47.869146 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:47.869155 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:47.869158 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:47.869206 (+ 48us) tablet.cc:698] Row locks acquired
0114 03:04:47.869208 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:04:47.869243 (+ 35us) write_op.cc:270] Start()
0114 03:04:47.876377 (+ 7134us) write_op.cc:276] Timestamp: P: 1736823887869206 usec, L: 0
0114 03:04:47.876380 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:47.876548 (+ 168us) log.cc:844] Serialized 354 byte log entry
0114 03:04:49.561621 (+1685073us) op_driver.cc:464] REPLICATION: finished
0114 03:04:49.561861 (+ 240us) write_op.cc:301] APPLY: starting
0114 03:04:49.561879 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:49.561913 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:49.561928 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:49.561984 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:49.562011 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:49.562015 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:49.562097 (+ 82us) log.cc:844] Serialized 43 byte log entry
0114 03:04:49.562315 (+ 218us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:49.562328 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:49.562334 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":180,"num_ops":1,"prepare.queue_time_us":63378,"prepare.run_cpu_time_us":609,"prepare.run_wall_time_us":7672,"replication_time_us":1685210,"thread_start_us":74,"threads_started":1}]]}
I20250114 03:04:49.620262 5176 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:49.620412 5176 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:49.620519 4771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.67:51049 (request call id 1919) took 1741 ms. Trace:
I20250114 03:04:49.620949 5176 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:49.621325 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:49.621347 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 20 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:49.621865 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:49.622007 5176 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:49.620625 4771 rpcz_store.cc:276] 0114 03:04:47.878633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:47.878695 (+ 62us) service_pool.cc:225] Handling call
0114 03:04:47.878864 (+ 169us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 03:04:47.878943 (+ 79us) raft_consensus.cc:1529] Early marking committed up to index 17
0114 03:04:47.879096 (+ 153us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 03:04:47.879199 (+ 103us) write_op.cc:270] Start()
0114 03:04:48.545345 (+666146us) write_op.cc:276] Timestamp: P: 1736823887078038 usec, L: 0
0114 03:04:48.545744 (+ 399us) write_op.cc:270] Start()
0114 03:04:48.895909 (+350165us) write_op.cc:276] Timestamp: P: 1736823887805367 usec, L: 0
0114 03:04:48.896196 (+ 287us) write_op.cc:270] Start()
0114 03:04:49.619375 (+723179us) write_op.cc:276] Timestamp: P: 1736823887869206 usec, L: 0
0114 03:04:49.619811 (+ 436us) log.cc:844] Serialized 1062 byte log entry
0114 03:04:49.619937 (+ 126us) raft_consensus.cc:1640] Marking committed up to 18
0114 03:04:49.620075 (+ 138us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:49.620108 (+ 33us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:49.620425 (+ 317us) raft_consensus.cc:1680] finished
0114 03:04:49.620426 (+ 1us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:49.620500 (+ 74us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:48.545727 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:48.545832 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:48.545837 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:48.545838 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:48.545906 (+ 68us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:48.545916 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:48.545918 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:48.545971 (+ 53us) tablet.cc:698] Row locks acquired
0114 03:04:48.545972 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:49.619983 (+1074011us) op_driver.cc:464] REPLICATION: finished
0114 03:04:49.620192 (+ 209us) write_op.cc:301] APPLY: starting
0114 03:04:49.620210 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:49.620239 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:49.620241 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:49.620312 (+ 71us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:49.620336 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:49.620359 (+ 23us) write_op.cc:312] APPLY: finished
0114 03:04:49.621039 (+ 680us) log.cc:844] Serialized 43 byte log entry
0114 03:04:49.621119 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:49.621130 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:49.621135 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:49.621225 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:49.621230 (+ 5us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:48.896150 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:48.896266 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:48.896271 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:48.896272 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:48.896341 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:48.896371 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:48.896373 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:48.896411 (+ 38us) tablet.cc:698] Row locks acquired
0114 03:04:48.896412 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:04:49.619734 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:49.619859 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:49.619864 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:49.619865 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:49.619934 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:49.619949 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:49.619954 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:49.619998 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:04:49.620000 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":168,"child_traces":[["op",{"apply.queue_time_us":170,"apply.run_cpu_time_us":419,"apply.run_wall_time_us":1020,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":239,"prepare.run_cpu_time_us":330,"prepare.run_wall_time_us":331,"replication_time_us":1740761,"thread_start_us":143,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":25,"prepare.queue_time_us":74,"prepare.run_cpu_time_us":325,"prepare.run_wall_time_us":326}],["op",{"num_ops":1,"partition_lock_wait_us":7,"prepare.queue_time_us":216,"prepare.run_cpu_time_us":361,"prepare.run_wall_time_us":361,"thread_start_us":78,"threads_started":1}]]}
I20250114 03:04:50.363473 5177 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:50.363605 5177 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:50.364120 5177 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:50.364552 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:50.364956 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:50.365363 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:50.365795 5221 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=15, attempt_no=0}) took 1753 ms. Trace:
I20250114 03:04:50.365870 5221 rpcz_store.cc:276] 0114 03:04:48.612370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:48.612427 (+ 57us) service_pool.cc:225] Handling call
0114 03:04:50.365782 (+1753355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:48.612926 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:48.613007 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:48.613010 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:48.613011 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:48.613062 (+ 51us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:48.613068 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:48.613070 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:48.613098 (+ 28us) tablet.cc:698] Row locks acquired
0114 03:04:48.613099 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:48.613125 (+ 26us) write_op.cc:270] Start()
0114 03:04:49.338283 (+725158us) write_op.cc:276] Timestamp: P: 1736823888613058 usec, L: 0
0114 03:04:49.338287 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:49.338493 (+ 206us) log.cc:844] Serialized 354 byte log entry
0114 03:04:50.364581 (+1026088us) op_driver.cc:464] REPLICATION: finished
0114 03:04:50.365427 (+ 846us) write_op.cc:301] APPLY: starting
0114 03:04:50.365448 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:50.365479 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:50.365494 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:50.365544 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:50.365567 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:50.365570 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:50.365635 (+ 65us) log.cc:844] Serialized 22 byte log entry
0114 03:04:50.365689 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:50.365700 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:50.365705 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":347,"num_ops":1,"prepare.queue_time_us":170,"prepare.run_cpu_time_us":546,"prepare.run_wall_time_us":725625,"replication_time_us":1026267,"thread_start_us":121,"threads_started":2}]]}
I20250114 03:04:50.472553 5176 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:50.472713 5176 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:50.473181 5176 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:50.473563 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:50.473655 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 21 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:50.474041 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:50.474195 5176 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:50.830261 5176 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:50.830402 5176 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:50.830925 5176 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:50.831267 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:50.831444 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 22 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:50.831890 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:50.831895 5221 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=16, attempt_no=0}) took 1268 ms. Trace:
I20250114 03:04:50.832119 5176 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:50.831979 5221 rpcz_store.cc:276] 0114 03:04:49.563819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:49.563902 (+ 83us) service_pool.cc:225] Handling call
0114 03:04:50.831883 (+1267981us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:49.564277 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:49.564362 (+ 85us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:49.564365 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:49.564366 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:49.564419 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:49.564424 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:49.564426 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:49.564457 (+ 31us) tablet.cc:698] Row locks acquired
0114 03:04:49.564458 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:49.564485 (+ 27us) write_op.cc:270] Start()
0114 03:04:50.098642 (+534157us) write_op.cc:276] Timestamp: P: 1736823889564427 usec, L: 0
0114 03:04:50.098646 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:50.098912 (+ 266us) log.cc:844] Serialized 354 byte log entry
0114 03:04:50.831193 (+732281us) op_driver.cc:464] REPLICATION: finished
0114 03:04:50.831281 (+ 88us) write_op.cc:301] APPLY: starting
0114 03:04:50.831300 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:50.831335 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:50.831348 (+ 13us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:50.831407 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:50.831434 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:50.831438 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:50.831511 (+ 73us) log.cc:844] Serialized 43 byte log entry
0114 03:04:50.831750 (+ 239us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:50.831776 (+ 26us) write_op.cc:454] Released schema lock
0114 03:04:50.831782 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":44,"num_ops":1,"prepare.queue_time_us":43,"prepare.run_cpu_time_us":632,"prepare.run_wall_time_us":534697,"replication_time_us":732514}]]}
I20250114 03:04:50.841943 5177 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:51.591939 5226 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:51.592080 5226 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:51.592717 5226 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:51.593001 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 23 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:51.593029 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 23 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:51.593504 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:51.593766 5226 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:51.594009 5229 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=17, attempt_no=0}) took 1227 ms. Trace:
I20250114 03:04:51.594105 5229 rpcz_store.cc:276] 0114 03:04:50.366955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:50.367013 (+ 58us) service_pool.cc:225] Handling call
0114 03:04:51.593996 (+1226983us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:50.367373 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:50.367456 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:50.367459 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:50.367460 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:50.367513 (+ 53us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:50.367519 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:50.367521 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:50.831844 (+464323us) lock_manager.cc:598] Waited 464172us for lock on <redacted> (key hash 2331656000287100367)
0114 03:04:50.831854 (+ 10us) tablet.cc:698] Row locks acquired
0114 03:04:50.831855 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:50.831896 (+ 41us) write_op.cc:270] Start()
0114 03:04:51.491059 (+659163us) write_op.cc:276] Timestamp: P: 1736823890831867 usec, L: 0
0114 03:04:51.491063 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:51.491330 (+ 267us) log.cc:844] Serialized 354 byte log entry
0114 03:04:51.592948 (+101618us) op_driver.cc:464] REPLICATION: finished
0114 03:04:51.593171 (+ 223us) write_op.cc:301] APPLY: starting
0114 03:04:51.593618 (+ 447us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:51.593644 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:51.593645 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:51.593704 (+ 59us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:51.593752 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:51.593756 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:51.593829 (+ 73us) log.cc:844] Serialized 43 byte log entry
0114 03:04:51.593884 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:51.593895 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:51.593900 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":174,"num_ops":1,"prepare.queue_time_us":45,"prepare.run_cpu_time_us":838,"prepare.run_wall_time_us":1124048,"raft.queue_time_us":74,"raft.run_cpu_time_us":350,"raft.run_wall_time_us":349,"replication_time_us":101855,"row_lock_wait_count":1,"row_lock_wait_us":464172,"spinlock_wait_cycles":190976,"thread_start_us":66,"threads_started":1}]]}
I20250114 03:04:51.640278 5226 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:51.640416 5226 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:51.640969 5226 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:51.641330 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 24 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:51.641367 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 24 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:51.641824 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:51.642097 5226 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:51.872285 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:51.872448 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:51.873042 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:51.873363 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 23 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:51.873489 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 23 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:51.873916 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:52.424252 5226 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:52.424403 5226 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:52.424954 5226 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:52.425355 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 25 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:52.425374 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 25 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:52.425815 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:52.426131 5226 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:52.715708 5226 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:52.715842 5226 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:52.716415 5226 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:52.716794 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 26 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:52.717075 5237 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=20, attempt_no=0}) took 1074 ms. Trace:
I20250114 03:04:52.717303 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 26 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:52.717144 5237 rpcz_store.cc:276] 0114 03:04:51.643090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:51.643165 (+ 75us) service_pool.cc:225] Handling call
0114 03:04:52.717058 (+1073893us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:51.794843 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:51.794957 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:51.794962 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:51.794964 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:51.795048 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:51.795057 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:51.795059 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:51.795095 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:04:51.795096 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:51.795132 (+ 36us) write_op.cc:270] Start()
0114 03:04:52.707262 (+912130us) write_op.cc:276] Timestamp: P: 1736823891795087 usec, L: 0
0114 03:04:52.707265 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:52.707416 (+ 151us) log.cc:844] Serialized 354 byte log entry
0114 03:04:52.716511 (+ 9095us) op_driver.cc:464] REPLICATION: finished
0114 03:04:52.716619 (+ 108us) write_op.cc:301] APPLY: starting
0114 03:04:52.716636 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:52.716667 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:52.716685 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:52.716745 (+ 60us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:52.716774 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:52.716778 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:52.716880 (+ 102us) log.cc:844] Serialized 43 byte log entry
0114 03:04:52.716930 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:52.716952 (+ 22us) write_op.cc:454] Released schema lock
0114 03:04:52.716957 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"num_ops":1,"prepare.queue_time_us":151371,"prepare.run_cpu_time_us":617,"prepare.run_wall_time_us":912683,"raft.queue_time_us":40,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":154,"replication_time_us":9213}]]}
I20250114 03:04:52.718115 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:52.718313 5226 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:53.345618 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:53.345927 4771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.67:51049 (request call id 2276) took 1472 ms. Trace:
I20250114 03:04:53.346006 4771 rpcz_store.cc:276] 0114 03:04:51.873752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:51.873804 (+ 52us) service_pool.cc:225] Handling call
0114 03:04:51.873944 (+ 140us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:51.874006 (+ 62us) raft_consensus.cc:1529] Early marking committed up to index 23
0114 03:04:51.874161 (+ 155us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:51.874267 (+ 106us) write_op.cc:270] Start()
0114 03:04:52.444407 (+570140us) write_op.cc:276] Timestamp: P: 1736823891491719 usec, L: 0
0114 03:04:52.444717 (+ 310us) write_op.cc:270] Start()
0114 03:04:53.344891 (+900174us) write_op.cc:276] Timestamp: P: 1736823891599301 usec, L: 0
0114 03:04:53.345306 (+ 415us) log.cc:844] Serialized 708 byte log entry
0114 03:04:53.345411 (+ 105us) raft_consensus.cc:1640] Marking committed up to 24
0114 03:04:53.345556 (+ 145us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:53.345579 (+ 23us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:53.345877 (+ 298us) raft_consensus.cc:1680] finished
0114 03:04:53.345878 (+ 1us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:53.345910 (+ 32us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:52.444725 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:52.444854 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:52.444858 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:52.444859 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:52.444935 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:52.444966 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:52.444969 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:52.445016 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:04:52.445017 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:53.345455 (+900438us) op_driver.cc:464] REPLICATION: finished
0114 03:04:53.345652 (+ 197us) write_op.cc:301] APPLY: starting
0114 03:04:53.345668 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:53.345700 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:53.345704 (+ 4us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:53.345757 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:53.345783 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:53.345801 (+ 18us) write_op.cc:312] APPLY: finished
0114 03:04:53.345856 (+ 55us) log.cc:844] Serialized 22 byte log entry
0114 03:04:53.345900 (+ 44us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:53.345910 (+ 10us) write_op.cc:454] Released schema lock
0114 03:04:53.345916 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:53.346002 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:53.346005 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:53.345213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:53.345313 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:53.345317 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:53.345318 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:53.345402 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:53.345410 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:53.345412 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:53.345458 (+ 46us) tablet.cc:698] Row locks acquired
0114 03:04:53.345460 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":56,"threads_started":1,"wal-append.queue_time_us":200,"child_traces":[["op",{"apply.queue_time_us":154,"apply.run_cpu_time_us":339,"apply.run_wall_time_us":338,"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":388,"prepare.run_wall_time_us":388,"replication_time_us":1471165,"thread_start_us":138,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":191,"prepare.run_cpu_time_us":341,"prepare.run_wall_time_us":340,"thread_start_us":86,"threads_started":1}]]}
I20250114 03:04:53.589108 5243 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:53.589246 5243 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:53.589771 5243 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:53.590166 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 27 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:53.590211 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 27 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:53.590598 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:53.590837 5243 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:53.590902 5246 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=21, attempt_no=0}) took 1163 ms. Trace:
I20250114 03:04:53.590984 5246 rpcz_store.cc:276] 0114 03:04:52.427128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:52.427190 (+ 62us) service_pool.cc:225] Handling call
0114 03:04:53.590889 (+1163699us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:52.707525 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:52.707647 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:52.707654 (+ 7us) write_op.cc:435] Acquired schema lock
0114 03:04:52.707655 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:52.707758 (+ 103us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:52.707769 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:52.707772 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:52.707819 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:04:52.707820 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:52.707857 (+ 37us) write_op.cc:270] Start()
0114 03:04:53.471992 (+764135us) write_op.cc:276] Timestamp: P: 1736823892707791 usec, L: 0
0114 03:04:53.471996 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:53.472145 (+ 149us) log.cc:844] Serialized 354 byte log entry
0114 03:04:53.590356 (+118211us) op_driver.cc:464] REPLICATION: finished
0114 03:04:53.590548 (+ 192us) write_op.cc:301] APPLY: starting
0114 03:04:53.590565 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:53.590594 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:53.590606 (+ 12us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:53.590656 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:53.590681 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:53.590686 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:53.590748 (+ 62us) log.cc:844] Serialized 22 byte log entry
0114 03:04:53.590803 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:53.590814 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:53.590819 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":140,"num_ops":1,"prepare.queue_time_us":280052,"prepare.run_cpu_time_us":605,"prepare.run_wall_time_us":764673,"replication_time_us":118330,"thread_start_us":55,"threads_started":1}]]}
I20250114 03:04:54.045152 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:54.045277 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:54.045886 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:54.046339 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 26 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:54.046576 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 26 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:54.046988 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:54.647174 5243 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:54.647351 5243 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:54.647927 5243 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:54.648308 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:54.648310 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 28 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:54.648813 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:54.649158 5255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=22, attempt_no=0}) took 1930 ms. Trace:
I20250114 03:04:54.649223 5255 rpcz_store.cc:276] 0114 03:04:52.718607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:52.718665 (+ 58us) service_pool.cc:225] Handling call
0114 03:04:54.649145 (+1930480us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:53.472221 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:53.472334 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:53.472339 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:53.472341 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:53.472419 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:53.472427 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:53.472429 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:53.472466 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:04:53.472467 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:53.472500 (+ 33us) write_op.cc:270] Start()
0114 03:04:53.780642 (+308142us) write_op.cc:276] Timestamp: P: 1736823893472453 usec, L: 0
0114 03:04:53.780647 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:53.780872 (+ 225us) log.cc:844] Serialized 354 byte log entry
0114 03:04:54.648467 (+867595us) op_driver.cc:464] REPLICATION: finished
0114 03:04:54.648743 (+ 276us) write_op.cc:301] APPLY: starting
0114 03:04:54.648761 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:54.648792 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:54.648807 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:54.648882 (+ 75us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:54.648915 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:54.648919 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:54.649004 (+ 85us) log.cc:844] Serialized 43 byte log entry
0114 03:04:54.649054 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:54.649065 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:54.649069 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":224,"num_ops":1,"prepare.queue_time_us":753270,"prepare.run_cpu_time_us":652,"prepare.run_wall_time_us":308705,"replication_time_us":867789,"thread_start_us":73,"threads_started":1}]]}
I20250114 03:04:55.089879 5243 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:55.091203 5255 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=23, attempt_no=0}) took 1498 ms. Trace:
I20250114 03:04:55.091284 5255 rpcz_store.cc:276] 0114 03:04:53.592290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:53.592349 (+ 59us) service_pool.cc:225] Handling call
0114 03:04:55.091189 (+1498840us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:53.780974 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:53.781089 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:53.781094 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:53.781097 (+ 3us) tablet.cc:659] Decoding operations
0114 03:04:53.781176 (+ 79us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:53.781186 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:53.781189 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:53.781234 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:04:53.781235 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:53.781271 (+ 36us) write_op.cc:270] Start()
0114 03:04:54.553410 (+772139us) write_op.cc:276] Timestamp: P: 1736823893781251 usec, L: 0
0114 03:04:54.553414 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:54.553609 (+ 195us) log.cc:844] Serialized 354 byte log entry
0114 03:04:55.090719 (+537110us) op_driver.cc:464] REPLICATION: finished
0114 03:04:55.090823 (+ 104us) write_op.cc:301] APPLY: starting
0114 03:04:55.090843 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:55.090883 (+ 40us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:55.090900 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:55.090954 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:55.090976 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:55.090979 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:04:55.091041 (+ 62us) log.cc:844] Serialized 43 byte log entry
0114 03:04:55.091089 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:55.091100 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:55.091103 (+ 3us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"num_ops":1,"prepare.queue_time_us":188328,"prepare.run_cpu_time_us":629,"prepare.run_wall_time_us":772693,"replication_time_us":537278}]]}
I20250114 03:04:55.317113 5243 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:55.317258 5243 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:55.317780 5243 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:55.318166 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 30 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:55.318163 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 30 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:55.318593 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:55.453547 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:55.453816 4771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.67:51049 (request call id 2278) took 1407 ms. Trace:
I20250114 03:04:55.453918 4771 rpcz_store.cc:276] 0114 03:04:54.046448 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:54.046568 (+ 120us) service_pool.cc:225] Handling call
0114 03:04:54.046764 (+ 196us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:54.046848 (+ 84us) raft_consensus.cc:1529] Early marking committed up to index 0
0114 03:04:54.046850 (+ 2us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:54.046915 (+ 65us) write_op.cc:270] Start()
0114 03:04:54.624047 (+577132us) write_op.cc:276] Timestamp: P: 1736823892707791 usec, L: 0
0114 03:04:54.624475 (+ 428us) write_op.cc:270] Start()
0114 03:04:55.452681 (+828206us) write_op.cc:276] Timestamp: P: 1736823893472453 usec, L: 0
0114 03:04:55.453188 (+ 507us) log.cc:844] Serialized 708 byte log entry
0114 03:04:55.453296 (+ 108us) raft_consensus.cc:1640] Marking committed up to 27
0114 03:04:55.453433 (+ 137us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:55.453470 (+ 37us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:55.453741 (+ 271us) raft_consensus.cc:1680] finished
0114 03:04:55.453744 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:55.453796 (+ 52us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:54.624446 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:54.624544 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:54.624548 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:54.624549 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:54.624613 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:54.624622 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:54.624624 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:54.624660 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:04:54.624661 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:55.453338 (+828677us) op_driver.cc:464] REPLICATION: finished
0114 03:04:55.453546 (+ 208us) write_op.cc:301] APPLY: starting
0114 03:04:55.453562 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:55.453594 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:55.453596 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:55.453650 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:55.453699 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:55.453730 (+ 31us) write_op.cc:312] APPLY: finished
0114 03:04:55.453802 (+ 72us) log.cc:844] Serialized 22 byte log entry
0114 03:04:55.453850 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:55.453858 (+ 8us) write_op.cc:454] Released schema lock
0114 03:04:55.453862 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:55.453922 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:55.453926 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:55.453136 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:55.453241 (+ 105us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:55.453245 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:55.453246 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:55.453313 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:55.453323 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:55.453325 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:55.453370 (+ 45us) tablet.cc:698] Row locks acquired
0114 03:04:55.453371 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":137,"child_traces":[["op",{"apply.queue_time_us":167,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":367,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":244,"prepare.run_cpu_time_us":324,"prepare.run_wall_time_us":325,"replication_time_us":1406402,"thread_start_us":145,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":291,"prepare.run_cpu_time_us":322,"prepare.run_wall_time_us":323,"thread_start_us":100,"threads_started":1}]]}
I20250114 03:04:56.241650 5243 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:56.243309 5264 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=25, attempt_no=0}) took 1150 ms. Trace:
I20250114 03:04:56.243388 5264 rpcz_store.cc:276] 0114 03:04:55.092529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:55.092597 (+ 68us) service_pool.cc:225] Handling call
0114 03:04:56.243292 (+1150695us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:55.093001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:55.093078 (+ 77us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:55.093081 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:55.093082 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:55.093131 (+ 49us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:55.093136 (+ 5us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:55.093138 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:55.093164 (+ 26us) tablet.cc:698] Row locks acquired
0114 03:04:55.093165 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:55.093208 (+ 43us) write_op.cc:270] Start()
0114 03:04:55.094310 (+ 1102us) write_op.cc:276] Timestamp: P: 1736823895093165 usec, L: 0
0114 03:04:55.094312 (+ 2us) op_driver.cc:348] REPLICATION: starting
0114 03:04:55.094422 (+ 110us) log.cc:844] Serialized 354 byte log entry
0114 03:04:56.242727 (+1148305us) op_driver.cc:464] REPLICATION: finished
0114 03:04:56.242881 (+ 154us) write_op.cc:301] APPLY: starting
0114 03:04:56.242901 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:56.242944 (+ 43us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:56.242964 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:56.243027 (+ 63us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:56.243053 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:56.243060 (+ 7us) write_op.cc:312] APPLY: finished
0114 03:04:56.243138 (+ 78us) log.cc:844] Serialized 43 byte log entry
0114 03:04:56.243188 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:56.243199 (+ 11us) write_op.cc:454] Released schema lock
0114 03:04:56.243204 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":119,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":401,"prepare.run_wall_time_us":1465,"replication_time_us":1148386,"thread_start_us":48,"threads_started":1}]]}
I20250114 03:04:56.650027 5243 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:56.650166 5243 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:56.650652 5243 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:56.651015 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:56.651046 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 32 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:56.651460 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:56.651510 5264 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=26, attempt_no=0}) took 1331 ms. Trace:
I20250114 03:04:56.651607 5243 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:56.651566 5264 rpcz_store.cc:276] 0114 03:04:55.320091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:55.320154 (+ 63us) service_pool.cc:225] Handling call
0114 03:04:56.651499 (+1331345us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:55.320524 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:55.320598 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:55.320600 (+ 2us) write_op.cc:435] Acquired schema lock
0114 03:04:55.320601 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:55.320647 (+ 46us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:55.320653 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:55.320654 (+ 1us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:55.320682 (+ 28us) tablet.cc:698] Row locks acquired
0114 03:04:55.320683 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:55.320707 (+ 24us) write_op.cc:270] Start()
0114 03:04:55.969882 (+649175us) write_op.cc:276] Timestamp: P: 1736823895320694 usec, L: 0
0114 03:04:55.969886 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:04:55.970090 (+ 204us) log.cc:844] Serialized 354 byte log entry
0114 03:04:56.650935 (+680845us) op_driver.cc:464] REPLICATION: finished
0114 03:04:56.651032 (+ 97us) write_op.cc:301] APPLY: starting
0114 03:04:56.651049 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:56.651080 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:56.651094 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:56.651224 (+ 130us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:56.651251 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:56.651255 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:56.651337 (+ 82us) log.cc:844] Serialized 43 byte log entry
0114 03:04:56.651392 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:56.651406 (+ 14us) write_op.cc:454] Released schema lock
0114 03:04:56.651411 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":55,"num_ops":1,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":540,"prepare.run_wall_time_us":649624,"replication_time_us":681006}]]}
I20250114 03:04:57.136863 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:57.137012 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:57.137108 4771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.67:51049 (request call id 2279) took 1681 ms. Trace:
I20250114 03:04:57.137524 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:57.137157 4771 rpcz_store.cc:276] 0114 03:04:55.455178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:55.455237 (+ 59us) service_pool.cc:225] Handling call
0114 03:04:55.455387 (+ 150us) raft_consensus.cc:1466] Updating replica for 3 ops
0114 03:04:55.455454 (+ 67us) raft_consensus.cc:1529] Early marking committed up to index 28
0114 03:04:55.455519 (+ 65us) raft_consensus.cc:1534] Triggering prepare for 3 ops
0114 03:04:55.455611 (+ 92us) write_op.cc:270] Start()
0114 03:04:55.972767 (+517156us) write_op.cc:276] Timestamp: P: 1736823893781251 usec, L: 0
0114 03:04:55.973145 (+ 378us) write_op.cc:270] Start()
0114 03:04:56.509306 (+536161us) write_op.cc:276] Timestamp: P: 1736823894651068 usec, L: 0
0114 03:04:56.509678 (+ 372us) write_op.cc:270] Start()
0114 03:04:57.135845 (+626167us) write_op.cc:276] Timestamp: P: 1736823895093165 usec, L: 0
0114 03:04:57.136349 (+ 504us) log.cc:844] Serialized 1062 byte log entry
0114 03:04:57.136487 (+ 138us) raft_consensus.cc:1640] Marking committed up to 30
0114 03:04:57.136787 (+ 300us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:57.136846 (+ 59us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:57.137036 (+ 190us) raft_consensus.cc:1680] finished
0114 03:04:57.137039 (+ 3us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:57.137092 (+ 53us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:55.973119 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:55.973219 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:55.973223 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:55.973224 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:55.973289 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:55.973298 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:55.973300 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:55.973339 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:04:55.973339 (+ 0us) write_op.cc:260] PREPARE: finished
0114 03:04:57.136535 (+1163196us) op_driver.cc:464] REPLICATION: finished
0114 03:04:57.136727 (+ 192us) write_op.cc:301] APPLY: starting
0114 03:04:57.136739 (+ 12us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:57.136767 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:57.136768 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:57.136842 (+ 74us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:57.136869 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:57.136893 (+ 24us) write_op.cc:312] APPLY: finished
0114 03:04:57.136971 (+ 78us) log.cc:844] Serialized 43 byte log entry
0114 03:04:57.137007 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:57.137016 (+ 9us) write_op.cc:454] Released schema lock
0114 03:04:57.137021 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:57.137113 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:57.137115 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:56.509680 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:56.509784 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:56.509788 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:56.509789 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:56.509853 (+ 64us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:56.509862 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:56.509864 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:56.509900 (+ 36us) tablet.cc:698] Row locks acquired
0114 03:04:56.509901 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:57.136675 (+626774us) op_driver.cc:464] REPLICATION: finished
0114 03:04:57.136892 (+ 217us) write_op.cc:301] APPLY: starting
0114 03:04:57.136901 (+ 9us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:57.136930 (+ 29us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:57.136932 (+ 2us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:57.136966 (+ 34us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:57.136985 (+ 19us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:57.136997 (+ 12us) write_op.cc:312] APPLY: finished
0114 03:04:57.137028 (+ 31us) log.cc:844] Serialized 22 byte log entry
0114 03:04:57.137047 (+ 19us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:57.137052 (+ 5us) write_op.cc:454] Released schema lock
0114 03:04:57.137055 (+ 3us) write_op.cc:341] FINISH: Updating metrics
0114 03:04:57.137113 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:57.137115 (+ 2us) write_op.cc:454] Released schema lock
Related trace 'op':
0114 03:04:57.136309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:57.136431 (+ 122us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:57.136435 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:57.136436 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:57.136523 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:57.136532 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:57.136534 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:57.136573 (+ 39us) tablet.cc:698] Row locks acquired
0114 03:04:57.136574 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":162,"child_traces":[["op",{"apply.queue_time_us":156,"apply.run_cpu_time_us":376,"apply.run_wall_time_us":375,"mutex_wait_us":27,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":192,"prepare.run_cpu_time_us":289,"prepare.run_wall_time_us":289,"replication_time_us":1680901,"thread_start_us":131,"threads_started":2}],["op",{"apply.queue_time_us":181,"apply.run_cpu_time_us":213,"apply.run_wall_time_us":213,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":236,"prepare.run_cpu_time_us":318,"prepare.run_wall_time_us":319,"replication_time_us":1163516,"spinlock_wait_cycles":15104,"thread_start_us":163,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":321,"prepare.run_cpu_time_us":322,"prepare.run_wall_time_us":323,"thread_start_us":98,"threads_started":1}]]}
I20250114 03:04:57.138028 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:57.138237 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 31 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:57.138583 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:57.138767 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:57.383265 5243 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:57.383404 5243 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:57.383894 5243 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:57.384424 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 33 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:57.384809 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 33 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:57.385278 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:57.385317 5273 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=27, attempt_no=0}) took 1140 ms. Trace:
I20250114 03:04:57.385382 5273 rpcz_store.cc:276] 0114 03:04:56.244631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:56.244698 (+ 67us) service_pool.cc:225] Handling call
0114 03:04:57.385303 (+1140605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:56.245079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:56.245152 (+ 73us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:56.245155 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:04:56.245156 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:56.245223 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:56.245229 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:56.245231 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:56.245258 (+ 27us) tablet.cc:698] Row locks acquired
0114 03:04:56.245259 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:56.245285 (+ 26us) write_op.cc:270] Start()
0114 03:04:56.984412 (+739127us) write_op.cc:276] Timestamp: P: 1736823896245227 usec, L: 0
0114 03:04:56.984415 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:56.984572 (+ 157us) log.cc:844] Serialized 354 byte log entry
0114 03:04:57.384606 (+400034us) op_driver.cc:464] REPLICATION: finished
0114 03:04:57.384918 (+ 312us) write_op.cc:301] APPLY: starting
0114 03:04:57.384937 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:57.384967 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:57.384981 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:57.385030 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:57.385058 (+ 28us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:57.385065 (+ 7us) write_op.cc:312] APPLY: finished
0114 03:04:57.385137 (+ 72us) log.cc:844] Serialized 43 byte log entry
0114 03:04:57.385201 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:57.385214 (+ 13us) write_op.cc:454] Released schema lock
0114 03:04:57.385220 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":257,"num_ops":1,"prepare.queue_time_us":39,"prepare.run_cpu_time_us":523,"prepare.run_wall_time_us":739568,"raft.queue_time_us":66,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":162,"replication_time_us":400160,"thread_start_us":84,"threads_started":1}]]}
I20250114 03:04:57.385772 5243 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:58.402570 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:58.402714 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:58.402920 4771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.67:51049 (request call id 2280) took 1264 ms. Trace:
I20250114 03:04:58.403180 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:58.402982 4771 rpcz_store.cc:276] 0114 03:04:57.138494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:57.138792 (+ 298us) service_pool.cc:225] Handling call
0114 03:04:57.138935 (+ 143us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:04:57.139000 (+ 65us) raft_consensus.cc:1529] Early marking committed up to index 31
0114 03:04:57.139062 (+ 62us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:04:57.139158 (+ 96us) write_op.cc:270] Start()
0114 03:04:57.846323 (+707165us) write_op.cc:276] Timestamp: P: 1736823895320694 usec, L: 0
0114 03:04:57.846678 (+ 355us) write_op.cc:270] Start()
0114 03:04:58.401834 (+555156us) write_op.cc:276] Timestamp: P: 1736823896245227 usec, L: 0
0114 03:04:58.402277 (+ 443us) log.cc:844] Serialized 708 byte log entry
0114 03:04:58.402385 (+ 108us) raft_consensus.cc:1640] Marking committed up to 32
0114 03:04:58.402523 (+ 138us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:04:58.402552 (+ 29us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:04:58.402837 (+ 285us) raft_consensus.cc:1680] finished
0114 03:04:58.402839 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:04:58.402894 (+ 55us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:57.846636 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:57.846747 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:57.846751 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:57.846753 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:57.846819 (+ 66us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:57.846829 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:57.846831 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:57.846866 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:57.846867 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:58.402422 (+555555us) op_driver.cc:464] REPLICATION: finished
0114 03:04:58.402641 (+ 219us) write_op.cc:301] APPLY: starting
0114 03:04:58.402657 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:58.402687 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:58.402688 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:58.402742 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:58.402765 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:58.402788 (+ 23us) write_op.cc:312] APPLY: finished
Related trace 'op':
0114 03:04:58.402162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:58.402279 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:58.402284 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:04:58.402285 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:58.402367 (+ 82us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:58.402379 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:58.402381 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:58.402428 (+ 47us) tablet.cc:698] Row locks acquired
0114 03:04:58.402429 (+ 1us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":61,"threads_started":1,"wal-append.queue_time_us":150,"child_traces":[["op",{"apply.queue_time_us":179,"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":192,"prepare.run_cpu_time_us":317,"prepare.run_wall_time_us":317,"replication_time_us":1263246,"thread_start_us":142,"threads_started":2}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":198,"prepare.run_cpu_time_us":361,"prepare.run_wall_time_us":362,"thread_start_us":77,"threads_started":1}]]}
I20250114 03:04:58.403650 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 33 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:58.403645 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 33 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:58.404073 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:58.404275 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:58.542049 5276 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:58.542214 5276 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:58.542773 5276 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:58.543164 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:58.543179 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:58.543644 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:58.543871 5276 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:58.544066 5283 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=28, attempt_no=0}) took 1891 ms. Trace:
I20250114 03:04:58.544140 5283 rpcz_store.cc:276] 0114 03:04:56.652966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:56.653038 (+ 72us) service_pool.cc:225] Handling call
0114 03:04:58.544043 (+1891005us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:56.984720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:56.984859 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:56.984865 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:56.984868 (+ 3us) tablet.cc:659] Decoding operations
0114 03:04:56.984951 (+ 83us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:56.984959 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:56.984962 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:56.984997 (+ 35us) tablet.cc:698] Row locks acquired
0114 03:04:56.984998 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:56.985035 (+ 37us) write_op.cc:270] Start()
0114 03:04:57.975211 (+990176us) write_op.cc:276] Timestamp: P: 1736823896985000 usec, L: 0
0114 03:04:57.975216 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:57.975422 (+ 206us) log.cc:844] Serialized 354 byte log entry
0114 03:04:58.543215 (+567793us) op_driver.cc:464] REPLICATION: finished
0114 03:04:58.543640 (+ 425us) write_op.cc:301] APPLY: starting
0114 03:04:58.543655 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:58.543683 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:58.543693 (+ 10us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:58.543742 (+ 49us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:58.543775 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:58.543780 (+ 5us) write_op.cc:312] APPLY: finished
0114 03:04:58.543866 (+ 86us) log.cc:844] Serialized 43 byte log entry
0114 03:04:58.543935 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:58.543947 (+ 12us) write_op.cc:454] Released schema lock
0114 03:04:58.543952 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":375,"num_ops":1,"prepare.queue_time_us":331373,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":990783,"raft.queue_time_us":39,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"replication_time_us":567969,"spinlock_wait_cycles":66432,"thread_start_us":81,"threads_started":1}]]}
I20250114 03:04:58.559377 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:58.559491 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:58.559931 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:58.560245 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:58.560257 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 34 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:58.560633 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:58.561242 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:58.849785 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:58.849941 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:58.850457 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:58.850788 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:58.850888 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:58.851286 5283 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=29, attempt_no=0}) took 1464 ms. Trace:
I20250114 03:04:58.851490 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:04:58.851357 5283 rpcz_store.cc:276] 0114 03:04:57.386861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:57.386921 (+ 60us) service_pool.cc:225] Handling call
0114 03:04:58.851275 (+1464354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:57.975528 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:57.975683 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:57.975689 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:04:57.975691 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:57.975779 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:57.975787 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:57.975789 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:57.975833 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:04:57.975834 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:57.975888 (+ 54us) write_op.cc:270] Start()
0114 03:04:58.812019 (+836131us) write_op.cc:276] Timestamp: P: 1736823897975839 usec, L: 0
0114 03:04:58.812022 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:04:58.812165 (+ 143us) log.cc:844] Serialized 354 byte log entry
0114 03:04:58.850747 (+ 38582us) op_driver.cc:464] REPLICATION: finished
0114 03:04:58.850839 (+ 92us) write_op.cc:301] APPLY: starting
0114 03:04:58.850875 (+ 36us) tablet.cc:1364] starting BulkCheckPresence
0114 03:04:58.850920 (+ 45us) tablet.cc:1367] finished BulkCheckPresence
0114 03:04:58.850936 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:04:58.850994 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:04:58.851021 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:04:58.851025 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:04:58.851096 (+ 71us) log.cc:844] Serialized 43 byte log entry
0114 03:04:58.851157 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:04:58.851169 (+ 12us) write_op.cc:454] Released schema lock
0114 03:04:58.851175 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34,"num_ops":1,"prepare.queue_time_us":588287,"prepare.run_cpu_time_us":652,"prepare.run_wall_time_us":836718,"raft.queue_time_us":243,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":343,"replication_time_us":38698,"spinlock_wait_cycles":93056}]]}
I20250114 03:04:58.851734 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:58.946604 5276 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:58.946790 5276 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:58.947335 5276 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:04:58.947750 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:58.947770 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 35 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:04:58.948182 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:04:59.405099 5276 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:04:59.836282 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:04:59.836432 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:04:59.837033 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:04:59.837358 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 36 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:04:59.837440 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 36 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:04:59.838265 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:00.247431 5276 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:00.247562 5276 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:00.248113 5276 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:00.248540 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 37 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:00.248530 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 37 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:00.248998 5293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=31, attempt_no=0}) took 1396 ms. Trace:
I20250114 03:05:00.249181 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:00.249075 5293 rpcz_store.cc:276] 0114 03:04:58.852692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:58.852755 (+ 63us) service_pool.cc:225] Handling call
0114 03:05:00.248984 (+1396229us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:58.862963 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:58.863077 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:58.863081 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:58.863082 (+ 1us) tablet.cc:659] Decoding operations
0114 03:04:58.863151 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:58.863159 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:58.863161 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:58.863194 (+ 33us) tablet.cc:698] Row locks acquired
0114 03:04:58.863195 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:58.863228 (+ 33us) write_op.cc:270] Start()
0114 03:04:59.373432 (+510204us) write_op.cc:276] Timestamp: P: 1736823898863162 usec, L: 0
0114 03:04:59.373437 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:04:59.373662 (+ 225us) log.cc:844] Serialized 354 byte log entry
0114 03:05:00.248395 (+874733us) op_driver.cc:464] REPLICATION: finished
0114 03:05:00.248580 (+ 185us) write_op.cc:301] APPLY: starting
0114 03:05:00.248596 (+ 16us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:00.248628 (+ 32us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:00.248643 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:00.248696 (+ 53us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:00.248722 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:00.248726 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:05:00.248787 (+ 61us) log.cc:844] Serialized 22 byte log entry
0114 03:05:00.248890 (+ 103us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:00.248901 (+ 11us) write_op.cc:454] Released schema lock
0114 03:05:00.248907 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":133,"num_ops":1,"prepare.queue_time_us":9852,"prepare.run_cpu_time_us":682,"prepare.run_wall_time_us":510761,"replication_time_us":874931,"thread_start_us":80,"threads_started":1}]]}
I20250114 03:05:00.426894 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:00.559171 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:00.559314 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:00.559931 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:00.560276 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 38 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:00.560464 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 38 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:00.560725 5293 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=32, attempt_no=0}) took 1152 ms. Trace:
I20250114 03:05:00.560974 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:00.560801 5293 rpcz_store.cc:276] 0114 03:04:59.407939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:04:59.407998 (+ 59us) service_pool.cc:225] Handling call
0114 03:05:00.560714 (+1152716us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:04:59.408348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:04:59.408446 (+ 98us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:04:59.408450 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:04:59.408452 (+ 2us) tablet.cc:659] Decoding operations
0114 03:04:59.408512 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:04:59.408519 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:04:59.408522 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:04:59.408554 (+ 32us) tablet.cc:698] Row locks acquired
0114 03:04:59.408555 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:04:59.408584 (+ 29us) write_op.cc:270] Start()
0114 03:05:00.140744 (+732160us) write_op.cc:276] Timestamp: P: 1736823899408569 usec, L: 0
0114 03:05:00.140748 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:05:00.140994 (+ 246us) log.cc:844] Serialized 354 byte log entry
0114 03:05:00.560183 (+419189us) op_driver.cc:464] REPLICATION: finished
0114 03:05:00.560264 (+ 81us) write_op.cc:301] APPLY: starting
0114 03:05:00.560283 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:00.560339 (+ 56us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:00.560358 (+ 19us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:00.560416 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:00.560442 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:00.560446 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:05:00.560525 (+ 79us) log.cc:844] Serialized 43 byte log entry
0114 03:05:00.560581 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:00.560593 (+ 12us) write_op.cc:454] Released schema lock
0114 03:05:00.560611 (+ 18us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"num_ops":1,"prepare.queue_time_us":50,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":732709,"replication_time_us":419401}]]}
I20250114 03:05:00.561203 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:01.245249 5276 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:01.288043 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:01.288208 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:01.288843 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:01.289197 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 39 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:01.289289 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 39 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:01.289788 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:01.289882 5301 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=33, attempt_no=0}) took 1039 ms. Trace:
I20250114 03:05:01.290051 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:01.289960 5301 rpcz_store.cc:276] 0114 03:05:00.250343 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:00.250401 (+ 58us) service_pool.cc:225] Handling call
0114 03:05:01.289867 (+1039466us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:00.250801 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:00.250880 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:00.250882 (+ 2us) write_op.cc:435] Acquired schema lock
0114 03:05:00.250883 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:00.250938 (+ 55us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:00.250944 (+ 6us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:00.250946 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:00.250975 (+ 29us) tablet.cc:698] Row locks acquired
0114 03:05:00.250975 (+ 0us) write_op.cc:260] PREPARE: finished
0114 03:05:00.251002 (+ 27us) write_op.cc:270] Start()
0114 03:05:00.953119 (+702117us) write_op.cc:276] Timestamp: P: 1736823900250939 usec, L: 0
0114 03:05:00.953124 (+ 5us) op_driver.cc:348] REPLICATION: starting
0114 03:05:00.953282 (+ 158us) log.cc:844] Serialized 354 byte log entry
0114 03:05:01.289127 (+335845us) op_driver.cc:464] REPLICATION: finished
0114 03:05:01.289424 (+ 297us) write_op.cc:301] APPLY: starting
0114 03:05:01.289441 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:01.289475 (+ 34us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:01.289490 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:01.289548 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:01.289596 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:01.289604 (+ 8us) write_op.cc:312] APPLY: finished
0114 03:05:01.289687 (+ 83us) log.cc:844] Serialized 43 byte log entry
0114 03:05:01.289750 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:01.289762 (+ 12us) write_op.cc:454] Released schema lock
0114 03:05:01.289769 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":246,"num_ops":1,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":480,"prepare.run_wall_time_us":702533,"replication_time_us":335972,"thread_start_us":79,"threads_started":1}]]}
I20250114 03:05:01.453359 5276 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:01.453516 5276 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:01.454038 5276 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:01.454423 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 39 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:01.454423 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 39 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:01.454845 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:01.791829 5276 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:01.793439 5304 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=34, attempt_no=0}) took 1231 ms. Trace:
I20250114 03:05:01.793524 5304 rpcz_store.cc:276] 0114 03:05:00.562296 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:00.562368 (+ 72us) service_pool.cc:225] Handling call
0114 03:05:01.793421 (+1231053us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:00.953369 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:00.953476 (+ 107us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:00.953480 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:05:00.953482 (+ 2us) tablet.cc:659] Decoding operations
0114 03:05:00.953557 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:00.953568 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:00.953573 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:00.953615 (+ 42us) tablet.cc:698] Row locks acquired
0114 03:05:00.953616 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:00.953653 (+ 37us) write_op.cc:270] Start()
0114 03:05:01.442834 (+489181us) write_op.cc:276] Timestamp: P: 1736823900953623 usec, L: 0
0114 03:05:01.442838 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:05:01.443020 (+ 182us) log.cc:844] Serialized 354 byte log entry
0114 03:05:01.792867 (+349847us) op_driver.cc:464] REPLICATION: finished
0114 03:05:01.793042 (+ 175us) write_op.cc:301] APPLY: starting
0114 03:05:01.793062 (+ 20us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:01.793095 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:01.793115 (+ 20us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:01.793169 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:01.793190 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:01.793193 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:05:01.793258 (+ 65us) log.cc:844] Serialized 43 byte log entry
0114 03:05:01.793310 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:01.793325 (+ 15us) write_op.cc:454] Released schema lock
0114 03:05:01.793330 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":115,"num_ops":1,"prepare.queue_time_us":390572,"prepare.run_cpu_time_us":652,"prepare.run_wall_time_us":489707,"replication_time_us":349970,"thread_start_us":59,"threads_started":1}]]}
I20250114 03:05:01.922446 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:01.922592 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:01.923058 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:01.923404 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 40 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:01.923544 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 40 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:01.923972 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:02.388315 5276 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:02.388460 5276 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:02.389123 5276 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:02.389343 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 41 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:02.389956 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:02.389971 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 41 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:02.390376 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:02.390666 5276 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:02.390802 5309 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=35, attempt_no=0}) took 1099 ms. Trace:
I20250114 03:05:02.390884 5309 rpcz_store.cc:276] 0114 03:05:01.291606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:01.291704 (+ 98us) service_pool.cc:225] Handling call
0114 03:05:02.390789 (+1099085us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:01.443111 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:01.443230 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:01.443235 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:05:01.443237 (+ 2us) tablet.cc:659] Decoding operations
0114 03:05:01.443306 (+ 69us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:01.443315 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:01.443317 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:01.443351 (+ 34us) tablet.cc:698] Row locks acquired
0114 03:05:01.443352 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:01.443393 (+ 41us) write_op.cc:270] Start()
0114 03:05:01.511565 (+ 68172us) write_op.cc:276] Timestamp: P: 1736823901443359 usec, L: 0
0114 03:05:01.511568 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:05:01.511789 (+ 221us) log.cc:844] Serialized 354 byte log entry
0114 03:05:02.389383 (+877594us) op_driver.cc:464] REPLICATION: finished
0114 03:05:02.390437 (+ 1054us) write_op.cc:301] APPLY: starting
0114 03:05:02.390452 (+ 15us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:02.390475 (+ 23us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:02.390487 (+ 12us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:02.390537 (+ 50us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:02.390561 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:02.390567 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:05:02.390635 (+ 68us) log.cc:844] Serialized 43 byte log entry
0114 03:05:02.390684 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:02.390697 (+ 13us) write_op.cc:454] Released schema lock
0114 03:05:02.390703 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1001,"num_ops":1,"prepare.queue_time_us":151048,"prepare.run_cpu_time_us":663,"prepare.run_wall_time_us":68739,"replication_time_us":877782,"thread_start_us":74,"threads_started":1}]]}
I20250114 03:05:02.681289 5276 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:02.681432 5276 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:02.681923 5276 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:02.682338 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:02.682339 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:02.682801 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:02.682976 5276 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:03.367374 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:03.367511 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:03.368041 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:03.368418 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:03.368453 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:03.368896 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:03.369174 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:03.405408 5314 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:03.405577 5314 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:03.406049 5314 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:03.406433 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:03.406433 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 42 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:03.406849 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:03.407162 5314 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:03.541847 5314 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:03.541978 5314 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:03.542529 5314 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:03.542969 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 43 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:03.542981 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 43 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:03.543458 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:03.543887 5314 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:03.544028 5316 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=37, attempt_no=0}) took 1151 ms. Trace:
I20250114 03:05:03.544086 5316 rpcz_store.cc:276] 0114 03:05:02.392277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:02.392317 (+ 40us) service_pool.cc:225] Handling call
0114 03:05:03.544014 (+1151697us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:02.658916 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:02.659030 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:02.659034 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:05:02.659035 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:02.659113 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:02.659126 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:02.659129 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:02.659171 (+ 42us) tablet.cc:698] Row locks acquired
0114 03:05:02.659172 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:02.659209 (+ 37us) write_op.cc:270] Start()
0114 03:05:03.517369 (+858160us) write_op.cc:276] Timestamp: P: 1736823902659165 usec, L: 0
0114 03:05:03.517373 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:05:03.517563 (+ 190us) log.cc:844] Serialized 354 byte log entry
0114 03:05:03.542774 (+ 25211us) op_driver.cc:464] REPLICATION: finished
0114 03:05:03.543044 (+ 270us) write_op.cc:301] APPLY: starting
0114 03:05:03.543063 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:03.543532 (+ 469us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:03.543548 (+ 16us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:03.543623 (+ 75us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:03.543665 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:03.543669 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:05:03.543732 (+ 63us) log.cc:844] Serialized 22 byte log entry
0114 03:05:03.543910 (+ 178us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:03.543922 (+ 12us) write_op.cc:454] Released schema lock
0114 03:05:03.543927 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":216,"num_ops":1,"prepare.queue_time_us":266223,"prepare.run_cpu_time_us":636,"prepare.run_wall_time_us":858705,"replication_time_us":25348,"spinlock_wait_cycles":19712,"thread_start_us":83,"threads_started":1}]]}
I20250114 03:05:04.505739 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:04.505926 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:04.506430 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:04.506842 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 43 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:04.506880 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 43 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:04.507318 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:04.890008 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:04.890169 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:04.890746 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:04.891180 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:04.891180 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 44 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:04.891597 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:04.891827 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:04.891988 5325 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=38, attempt_no=0}) took 2207 ms. Trace:
I20250114 03:05:04.892058 5325 rpcz_store.cc:276] 0114 03:05:02.684313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:02.684382 (+ 69us) service_pool.cc:225] Handling call
0114 03:05:04.891970 (+2207588us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:03.517653 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:03.517763 (+ 110us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:03.517769 (+ 6us) write_op.cc:435] Acquired schema lock
0114 03:05:03.517771 (+ 2us) tablet.cc:659] Decoding operations
0114 03:05:03.517847 (+ 76us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:03.517858 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:03.517861 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:03.517920 (+ 59us) tablet.cc:698] Row locks acquired
0114 03:05:03.517922 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:05:03.517959 (+ 37us) write_op.cc:270] Start()
0114 03:05:04.175122 (+657163us) write_op.cc:276] Timestamp: P: 1736823903517924 usec, L: 0
0114 03:05:04.175126 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:05:04.175306 (+ 180us) log.cc:844] Serialized 354 byte log entry
0114 03:05:04.891191 (+715885us) op_driver.cc:464] REPLICATION: finished
0114 03:05:04.891400 (+ 209us) write_op.cc:301] APPLY: starting
0114 03:05:04.891417 (+ 17us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:04.891448 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:04.891466 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:04.891524 (+ 58us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:04.891550 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:04.891597 (+ 47us) write_op.cc:312] APPLY: finished
0114 03:05:04.891688 (+ 91us) log.cc:844] Serialized 43 byte log entry
0114 03:05:04.891863 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:04.891875 (+ 12us) write_op.cc:454] Released schema lock
0114 03:05:04.891880 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":163,"num_ops":1,"prepare.queue_time_us":832990,"prepare.run_cpu_time_us":617,"prepare.run_wall_time_us":657718,"replication_time_us":716036,"thread_start_us":58,"threads_started":1}]]}
I20250114 03:05:05.694960 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:05.695094 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:05.695647 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:05.696055 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:05.696058 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 45 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:05.696509 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:05.696712 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:05.696799 5332 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=39, attempt_no=0}) took 2151 ms. Trace:
I20250114 03:05:05.696944 5332 rpcz_store.cc:276] 0114 03:05:03.545276 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:03.545333 (+ 57us) service_pool.cc:225] Handling call
0114 03:05:05.696787 (+2151454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:04.175412 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:04.175528 (+ 116us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:04.175533 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:05:04.175534 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:04.175614 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:04.175626 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:04.175631 (+ 5us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:04.175681 (+ 50us) tablet.cc:698] Row locks acquired
0114 03:05:04.175682 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:04.175721 (+ 39us) write_op.cc:270] Start()
0114 03:05:04.389881 (+214160us) write_op.cc:276] Timestamp: P: 1736823904175681 usec, L: 0
0114 03:05:04.389885 (+ 4us) op_driver.cc:348] REPLICATION: starting
0114 03:05:04.390098 (+ 213us) log.cc:844] Serialized 354 byte log entry
0114 03:05:05.696172 (+1306074us) op_driver.cc:464] REPLICATION: finished
0114 03:05:05.696384 (+ 212us) write_op.cc:301] APPLY: starting
0114 03:05:05.696403 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:05.696434 (+ 31us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:05.696449 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:05.696504 (+ 55us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:05.696528 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:05.696532 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:05:05.696621 (+ 89us) log.cc:844] Serialized 43 byte log entry
0114 03:05:05.696676 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:05.696688 (+ 12us) write_op.cc:454] Released schema lock
0114 03:05:05.696705 (+ 17us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":158,"num_ops":1,"prepare.queue_time_us":629772,"prepare.run_cpu_time_us":674,"prepare.run_wall_time_us":214745,"replication_time_us":1306255,"thread_start_us":66,"threads_started":1}]]}
I20250114 03:05:05.781116 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:05.781550 4771 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.1.109.67:51049 (request call id 3128) took 1274 ms. Trace:
I20250114 03:05:05.781617 4771 rpcz_store.cc:276] 0114 03:05:04.507181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:04.507225 (+ 44us) service_pool.cc:225] Handling call
0114 03:05:04.507367 (+ 142us) raft_consensus.cc:1466] Updating replica for 2 ops
0114 03:05:04.507449 (+ 82us) raft_consensus.cc:1529] Early marking committed up to index 43
0114 03:05:04.507590 (+ 141us) raft_consensus.cc:1534] Triggering prepare for 2 ops
0114 03:05:04.507692 (+ 102us) write_op.cc:270] Start()
0114 03:05:05.043880 (+536188us) write_op.cc:276] Timestamp: P: 1736823903517924 usec, L: 0
0114 03:05:05.044219 (+ 339us) write_op.cc:270] Start()
0114 03:05:05.780379 (+736160us) write_op.cc:276] Timestamp: P: 1736823904175681 usec, L: 0
0114 03:05:05.780878 (+ 499us) log.cc:844] Serialized 708 byte log entry
0114 03:05:05.780991 (+ 113us) raft_consensus.cc:1640] Marking committed up to 43
0114 03:05:05.780999 (+ 8us) raft_consensus.cc:1692] Filling consensus response to leader.
0114 03:05:05.781048 (+ 49us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0114 03:05:05.781489 (+ 441us) raft_consensus.cc:1680] finished
0114 03:05:05.781491 (+ 2us) raft_consensus.cc:1686] UpdateReplicas() finished
0114 03:05:05.781533 (+ 42us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:05.044208 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:05.044311 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:05.044316 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:05:05.044317 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:05.044382 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:05.044391 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:05.044393 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:05.044430 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:05:05.044431 (+ 1us) write_op.cc:260] PREPARE: finished
Related trace 'op':
0114 03:05:05.780770 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:05.780908 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:05.780913 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:05:05.780914 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:05.780984 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:05.780994 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:05.780997 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:05.781056 (+ 59us) tablet.cc:698] Row locks acquired
0114 03:05:05.781058 (+ 2us) write_op.cc:260] PREPARE: finished
Metrics: {"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":149,"child_traces":[["op",{"num_ops":1,"partition_lock_wait_us":5,"prepare.queue_time_us":194,"prepare.run_cpu_time_us":295,"prepare.run_wall_time_us":296,"thread_start_us":79,"threads_started":1}],["op",{"num_ops":1,"partition_lock_wait_us":6,"prepare.queue_time_us":212,"prepare.run_cpu_time_us":392,"prepare.run_wall_time_us":412,"thread_start_us":92,"threads_started":1}]]}
I20250114 03:05:05.804787 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:05.804946 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:05.805575 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:05.806001 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:05.806001 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:05.806419 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:05.806607 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:05.935977 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:05.936127 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:05.936687 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:05.936976 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:05.937077 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:05.938006 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:05.938159 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:05.994465 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:05.994606 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:05.995023 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:05.995337 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:05.995440 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:05.995841 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:05.996058 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:06.004554 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:06.004665 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:06.005123 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:06.005435 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:06.005988 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 46 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:06.006321 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:06.006525 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:06.386068 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:06.386236 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:06.386893 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:06.387396 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 47 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:06.387396 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 47 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:06.387852 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:06.847894 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:06.848037 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:06.848682 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:06.849155 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 47 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:06.849160 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 47 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:06.849640 4704 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:07.029479 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:07.030963 5345 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=42, attempt_no=0}) took 1223 ms. Trace:
I20250114 03:05:07.031037 5345 rpcz_store.cc:276] 0114 03:05:05.807592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:05.807656 (+ 64us) service_pool.cc:225] Handling call
0114 03:05:07.030923 (+1223267us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:06.033170 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:06.033276 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:06.033281 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:05:06.033282 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:06.033356 (+ 74us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:06.033364 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:06.033366 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:06.033410 (+ 44us) tablet.cc:698] Row locks acquired
0114 03:05:06.033413 (+ 3us) write_op.cc:260] PREPARE: finished
0114 03:05:06.033455 (+ 42us) write_op.cc:270] Start()
0114 03:05:06.043611 (+ 10156us) write_op.cc:276] Timestamp: P: 1736823906033382 usec, L: 0
0114 03:05:06.043614 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:05:06.043758 (+ 144us) log.cc:844] Serialized 354 byte log entry
0114 03:05:07.030337 (+986579us) op_driver.cc:464] REPLICATION: finished
0114 03:05:07.030519 (+ 182us) write_op.cc:301] APPLY: starting
0114 03:05:07.030537 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:07.030572 (+ 35us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:07.030589 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:07.030646 (+ 57us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:07.030668 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:07.030671 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:05:07.030743 (+ 72us) log.cc:844] Serialized 43 byte log entry
0114 03:05:07.030797 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:07.030809 (+ 12us) write_op.cc:454] Released schema lock
0114 03:05:07.030813 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":127,"num_ops":1,"prepare.queue_time_us":225196,"prepare.run_cpu_time_us":556,"prepare.run_wall_time_us":10639,"replication_time_us":986696,"thread_start_us":65,"threads_started":1}]]}
I20250114 03:05:07.268872 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:07.269030 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:07.269490 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:07.269882 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:07.269930 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:07.270332 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:07.270545 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:07.412555 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:07.462963 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:07.463111 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:07.463593 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:07.463882 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:07.463950 4902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 49 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:07.464314 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:07.464452 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:07.955121 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:07.955260 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:07.955755 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:07.956126 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 50 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:07.956282 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 50 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:07.956766 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:07.957036 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:07.979039 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:07.979187 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:07.979836 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:07.980180 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 50 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:07.980163 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 50 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:07.980572 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:07.981009 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:08.523190 5227 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:08.523345 5227 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:08.523877 5227 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225), 77a32f4dd261435cb6dc91526239a9cd (127.1.109.66:37591)
I20250114 03:05:08.524240 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:08.524334 4901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "f6f5a4e02dd1463db75006734f788a32" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "77a32f4dd261435cb6dc91526239a9cd" is_pre_election: true
I20250114 03:05:08.524809 4705 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f6f5a4e02dd1463db75006734f788a32; no voters: 77a32f4dd261435cb6dc91526239a9cd, 879969dde1ff460bb29cf9319dcd8746
I20250114 03:05:08.524921 5354 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38322 (ReqId={client: a85db9ca66b844b18a3f9193c25e6d55, seq_no=45, attempt_no=0}) took 1253 ms. Trace:
I20250114 03:05:08.525107 5227 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P f6f5a4e02dd1463db75006734f788a32 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:08.525027 5354 rpcz_store.cc:276] 0114 03:05:07.271563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:07.271622 (+ 59us) service_pool.cc:225] Handling call
0114 03:05:08.524904 (+1253282us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:07.647842 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 82ae6c5627ba43f1b3a5ecf476539b81
0114 03:05:07.647950 (+ 108us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:07.647955 (+ 5us) write_op.cc:435] Acquired schema lock
0114 03:05:07.647957 (+ 2us) tablet.cc:659] Decoding operations
0114 03:05:07.648054 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:07.648065 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:07.648068 (+ 3us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:07.956806 (+308738us) lock_manager.cc:598] Waited 308674us for lock on <redacted> (key hash 11785448468237383701)
0114 03:05:07.956843 (+ 37us) tablet.cc:698] Row locks acquired
0114 03:05:07.956845 (+ 2us) write_op.cc:260] PREPARE: finished
0114 03:05:07.956923 (+ 78us) write_op.cc:270] Start()
0114 03:05:08.027080 (+ 70157us) write_op.cc:276] Timestamp: P: 1736823907956865 usec, L: 0
0114 03:05:08.027083 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:05:08.027236 (+ 153us) log.cc:844] Serialized 354 byte log entry
0114 03:05:08.524100 (+496864us) op_driver.cc:464] REPLICATION: finished
0114 03:05:08.524303 (+ 203us) write_op.cc:301] APPLY: starting
0114 03:05:08.524336 (+ 33us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:08.524369 (+ 33us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:08.524370 (+ 1us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:08.524426 (+ 56us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:08.524452 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:08.524456 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:05:08.524528 (+ 72us) log.cc:844] Serialized 43 byte log entry
0114 03:05:08.524758 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:08.524774 (+ 16us) write_op.cc:454] Released schema lock
0114 03:05:08.524779 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":146,"num_ops":1,"prepare.queue_time_us":375895,"prepare.run_cpu_time_us":723,"prepare.run_wall_time_us":379458,"replication_time_us":496987,"row_lock_wait_count":1,"row_lock_wait_us":308674,"spinlock_wait_cycles":75136,"thread_start_us":72,"threads_started":1}]]}
I20250114 03:05:08.592258 4751 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:08.594064 5014 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:05:09.022329 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:09.022468 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:09.022964 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:09.023849 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:09.024803 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:09.025190 4835 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:09.025445 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20250114 03:05:09.025571 4882 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:09.032900 5318 raft_consensus.cc:491] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 879969dde1ff460bb29cf9319dcd8746)
I20250114 03:05:09.033028 5318 raft_consensus.cc:513] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f5a4e02dd1463db75006734f788a32" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 37273 } } peers { permanent_uuid: "879969dde1ff460bb29cf9319dcd8746" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 44225 } } peers { permanent_uuid: "77a32f4dd261435cb6dc91526239a9cd" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 37591 } }
I20250114 03:05:09.033430 5318 leader_election.cc:290] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273), 879969dde1ff460bb29cf9319dcd8746 (127.1.109.67:44225)
I20250114 03:05:09.033674 4771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "f6f5a4e02dd1463db75006734f788a32" is_pre_election: true
I20250114 03:05:09.033843 5034 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82ae6c5627ba43f1b3a5ecf476539b81" candidate_uuid: "77a32f4dd261435cb6dc91526239a9cd" candidate_term: 9 candidate_status { last_received { term: 8 index: 51 } } ignore_live_leader: false dest_uuid: "879969dde1ff460bb29cf9319dcd8746" is_pre_election: true
I20250114 03:05:09.034260 4837 leader_election.cc:304] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 77a32f4dd261435cb6dc91526239a9cd; no voters: 879969dde1ff460bb29cf9319dcd8746, f6f5a4e02dd1463db75006734f788a32
I20250114 03:05:09.034451 5318 raft_consensus.cc:2743] T 82ae6c5627ba43f1b3a5ecf476539b81 P 77a32f4dd261435cb6dc91526239a9cd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
aa45548db049458cb0ae92a9bdf3d484 | 127.1.109.126:34547 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:43617 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
77a32f4dd261435cb6dc91526239a9cd | 127.1.109.66:37591 | HEALTHY | <none> | 0 | 0
879969dde1ff460bb29cf9319dcd8746 | 127.1.109.67:44225 | HEALTHY | <none> | 1 | 0
f6f5a4e02dd1463db75006734f788a32 | 127.1.109.65:37273 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
--------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
inject_latency_ms_before_starting_op | 1000 | hidden,advanced | all 3 server(s) checked
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:37273
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:37591
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:44225
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:37273
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:37591
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:44225
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:43617 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
---------------+----+---------+---------------+---------+------------+------------------+-------------
test-workload | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250114 03:05:09.050813 1461 log_verifier.cc:126] Checking tablet 82ae6c5627ba43f1b3a5ecf476539b81
I20250114 03:05:09.090802 1461 log_verifier.cc:177] Verified matching terms for 51 ops in tablet 82ae6c5627ba43f1b3a5ecf476539b81
I20250114 03:05:09.114940 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 4689
W20250114 03:05:09.123428 4968 connection.cc:537] client connection to 127.1.109.65:37273 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:05:09.123567 4968 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20250114 03:05:09.123786 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 4820
W20250114 03:05:09.125792 4968 consensus_peers.cc:487] T 82ae6c5627ba43f1b3a5ecf476539b81 P 879969dde1ff460bb29cf9319dcd8746 -> Peer f6f5a4e02dd1463db75006734f788a32 (127.1.109.65:37273): Couldn't send request to peer f6f5a4e02dd1463db75006734f788a32. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.65:37273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 03:05:09.130720 4967 connection.cc:537] client connection to 127.1.109.66:37591 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 03:05:09.131064 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 4951
I20250114 03:05:09.137423 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 4598
2025-01-14T03:05:09Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ChurnyElectionsWithDuplicateKeys (32581 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElection
I20250114 03:05:09.161497 1461 test_util.cc:274] Using random seed: -735798765
I20250114 03:05:09.162276 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:09.162321 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:09.162333 1461 ts_itest-base.cc:114] 5 tablet servers
I20250114 03:05:09.162343 1461 ts_itest-base.cc:115] 5 replicas per TS
I20250114 03:05:09.162353 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:09Z Disabled control of system clock
I20250114 03:05:09.170253 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:42231
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:42231 with env {}
W20250114 03:05:09.279198 5404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:09.279464 5404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:09.279541 5404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:09.282829 5404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:09.282920 5404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:09.282958 5404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:09.282989 5404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:09.286870 5404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:42231
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:42231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:09.288036 5404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:09.289309 5404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:09.297636 5410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:09.298390 5404 server_base.cc:1034] running on GCE node
W20250114 03:05:09.299288 5412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.300067 5409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:09.300645 5404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:09.301779 5404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:09.302974 5404 hybrid_clock.cc:648] HybridClock initialized: now 1736823909302956 us; error 31 us; skew 500 ppm
I20250114 03:05:09.305227 5404 webserver.cc:458] Webserver started at http://127.1.109.126:40563/ using document root <none> and password file <none>
I20250114 03:05:09.305851 5404 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:09.305917 5404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:09.306100 5404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:09.307864 5404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0316a506daeb475f957ba2e84cee9332"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.308369 5404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0316a506daeb475f957ba2e84cee9332"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.312065 5404 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:05:09.314364 5418 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.315424 5404 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:09.315541 5404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0316a506daeb475f957ba2e84cee9332"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.315722 5404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:09.328491 5404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:09.329188 5404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:09.329358 5404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:09.336724 5404 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:42231
I20250114 03:05:09.336731 5470 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:42231 every 8 connection(s)
I20250114 03:05:09.337824 5404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:09.340423 5471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:09.345679 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 5404
I20250114 03:05:09.345697 5471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332: Bootstrap starting.
I20250114 03:05:09.345775 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:09.348132 5471 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:09.348904 5471 log.cc:826] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:09.350615 5471 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332: No bootstrap required, opened a new log
I20250114 03:05:09.353901 5471 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0316a506daeb475f957ba2e84cee9332" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 42231 } }
I20250114 03:05:09.354126 5471 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:09.354184 5471 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0316a506daeb475f957ba2e84cee9332, State: Initialized, Role: FOLLOWER
I20250114 03:05:09.354642 5471 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0316a506daeb475f957ba2e84cee9332" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 42231 } }
I20250114 03:05:09.354787 5471 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:09.354863 5471 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:09.354979 5471 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:09.355854 5471 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0316a506daeb475f957ba2e84cee9332" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 42231 } }
I20250114 03:05:09.356169 5471 leader_election.cc:304] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0316a506daeb475f957ba2e84cee9332; no voters:
I20250114 03:05:09.356431 5471 leader_election.cc:290] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:09.356603 5476 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:09.356936 5476 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [term 1 LEADER]: Becoming Leader. State: Replica: 0316a506daeb475f957ba2e84cee9332, State: Running, Role: LEADER
I20250114 03:05:09.357285 5476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0316a506daeb475f957ba2e84cee9332" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 42231 } }
I20250114 03:05:09.357534 5471 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:09.358736 5478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0316a506daeb475f957ba2e84cee9332. Latest consensus state: current_term: 1 leader_uuid: "0316a506daeb475f957ba2e84cee9332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0316a506daeb475f957ba2e84cee9332" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 42231 } } }
I20250114 03:05:09.358855 5478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:09.359201 5477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0316a506daeb475f957ba2e84cee9332" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0316a506daeb475f957ba2e84cee9332" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 42231 } } }
I20250114 03:05:09.359261 5485 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:09.359329 5477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:09.362612 5485 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:09.367960 5485 catalog_manager.cc:1348] Generated new cluster ID: 90072c81bc9748b5a4246f1cadbf25b9
I20250114 03:05:09.368036 5485 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:09.389032 5485 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:09.390040 5485 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:09.398511 5485 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 0316a506daeb475f957ba2e84cee9332: Generated new TSK 0
I20250114 03:05:09.399246 5485 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:09.404517 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20250114 03:05:09.516947 5495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:09.517184 5495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:09.517256 5495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:09.520539 5495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:09.520689 5495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:09.524565 5495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.5495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:09.525734 5495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:09.526876 5495 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:09.535482 5503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.536108 5501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.537084 5500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:09.537922 5495 server_base.cc:1034] running on GCE node
I20250114 03:05:09.538336 5495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:09.538981 5495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:09.540169 5495 hybrid_clock.cc:648] HybridClock initialized: now 1736823909540144 us; error 36 us; skew 500 ppm
I20250114 03:05:09.542215 5495 webserver.cc:458] Webserver started at http://127.1.109.65:36385/ using document root <none> and password file <none>
I20250114 03:05:09.542811 5495 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:09.542876 5495 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:09.543056 5495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:09.544775 5495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9ea06074428644ac86b5a52eab7554e6"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.545295 5495 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9ea06074428644ac86b5a52eab7554e6"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.548772 5495 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20250114 03:05:09.551036 5509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.552188 5495 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:09.552318 5495 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9ea06074428644ac86b5a52eab7554e6"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.552423 5495 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:09.566161 5495 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:09.566844 5495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:09.567018 5495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:09.567564 5495 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:09.568476 5495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:09.568523 5495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.568580 5495 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:09.568611 5495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.577181 5495 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:42579
I20250114 03:05:09.577250 5622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:42579 every 8 connection(s)
I20250114 03:05:09.578194 5495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:09.581255 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 5495
I20250114 03:05:09.581373 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:09.584000 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250114 03:05:09.588392 5623 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:09.588670 5623 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:09.589288 5623 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:09.590893 5435 ts_manager.cc:194] Registered new tserver with Master: 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579)
I20250114 03:05:09.592067 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:54451
W20250114 03:05:09.694288 5626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:09.694610 5626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:09.694681 5626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:09.697857 5626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:09.698006 5626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:09.701790 5626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.5626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:09.702908 5626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:09.704093 5626 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:09.714804 5632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.715536 5631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.716459 5634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:09.717379 5626 server_base.cc:1034] running on GCE node
I20250114 03:05:09.717900 5626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:09.718657 5626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:09.719871 5626 hybrid_clock.cc:648] HybridClock initialized: now 1736823909719815 us; error 61 us; skew 500 ppm
I20250114 03:05:09.722452 5626 webserver.cc:458] Webserver started at http://127.1.109.66:45627/ using document root <none> and password file <none>
I20250114 03:05:09.723070 5626 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:09.723125 5626 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:09.723300 5626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:09.725118 5626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "44a39e7d733b422e8db32bae91ff0d56"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.725608 5626 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "44a39e7d733b422e8db32bae91ff0d56"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.729146 5626 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20250114 03:05:09.731535 5640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.732774 5626 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:09.732932 5626 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "44a39e7d733b422e8db32bae91ff0d56"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.733038 5626 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:09.756143 5626 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:09.756944 5626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:09.757129 5626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:09.757742 5626 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:09.758709 5626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:09.758757 5626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.758834 5626 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:09.758862 5626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.768275 5626 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:34653
I20250114 03:05:09.768301 5753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:34653 every 8 connection(s)
I20250114 03:05:09.769356 5626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:09.773031 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 5626
I20250114 03:05:09.773160 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:09.775559 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250114 03:05:09.779561 5754 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:09.779808 5754 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:09.780381 5754 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:09.781507 5435 ts_manager.cc:194] Registered new tserver with Master: 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653)
I20250114 03:05:09.782126 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:46675
W20250114 03:05:09.889107 5757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:09.889365 5757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:09.889447 5757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:09.893018 5757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:09.893182 5757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:09.897238 5757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.5757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:09.898399 5757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:09.899652 5757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:09.908607 5765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.909443 5763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:09.910319 5762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:09.911338 5757 server_base.cc:1034] running on GCE node
I20250114 03:05:09.911751 5757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:09.912339 5757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:09.913504 5757 hybrid_clock.cc:648] HybridClock initialized: now 1736823909913468 us; error 48 us; skew 500 ppm
I20250114 03:05:09.915421 5757 webserver.cc:458] Webserver started at http://127.1.109.67:41565/ using document root <none> and password file <none>
I20250114 03:05:09.915989 5757 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:09.916041 5757 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:09.916188 5757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:09.918007 5757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1fc998e25ad148bb9d6245ba68a93b63"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.918503 5757 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1fc998e25ad148bb9d6245ba68a93b63"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.921916 5757 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:05:09.924054 5771 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.925227 5757 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:09.925336 5757 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1fc998e25ad148bb9d6245ba68a93b63"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:09.925431 5757 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:09.937281 5757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:09.938030 5757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:09.938207 5757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:09.938799 5757 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:09.939713 5757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:09.939760 5757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.939833 5757 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:09.939860 5757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:09.948936 5757 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:38577
I20250114 03:05:09.948969 5884 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:38577 every 8 connection(s)
I20250114 03:05:09.949963 5757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:09.952255 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 5757
I20250114 03:05:09.952349 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:09.954627 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.68:0
--local_ip_for_outbound_sockets=127.1.109.68
--webserver_interface=127.1.109.68
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250114 03:05:09.960036 5885 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:09.960274 5885 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:09.960917 5885 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:09.962066 5435 ts_manager.cc:194] Registered new tserver with Master: 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:09.962651 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:45923
W20250114 03:05:10.064357 5888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:10.064643 5888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:10.064725 5888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:10.068337 5888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:10.068504 5888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.68
I20250114 03:05:10.072292 5888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.1.109.68
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.5888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.68
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:10.073644 5888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:10.074925 5888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:10.083914 5896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:10.084594 5894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:10.085397 5893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:10.086416 5888 server_base.cc:1034] running on GCE node
I20250114 03:05:10.086861 5888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:10.087467 5888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:10.088639 5888 hybrid_clock.cc:648] HybridClock initialized: now 1736823910088603 us; error 49 us; skew 500 ppm
I20250114 03:05:10.090787 5888 webserver.cc:458] Webserver started at http://127.1.109.68:43873/ using document root <none> and password file <none>
I20250114 03:05:10.091424 5888 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:10.091486 5888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:10.091706 5888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:10.093572 5888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "0ca84a15c67f41998b81c4543d8c67f1"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:10.094059 5888 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "0ca84a15c67f41998b81c4543d8c67f1"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:10.097630 5888 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:05:10.099864 5902 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:10.100929 5888 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:10.101033 5888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "0ca84a15c67f41998b81c4543d8c67f1"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:10.101121 5888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:10.115927 5888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:10.116633 5888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:10.116878 5888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:10.117482 5888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:10.118438 5888 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:10.118500 5888 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:10.118564 5888 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:10.118599 5888 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:10.128157 5888 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.68:34193
I20250114 03:05:10.128245 6015 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.68:34193 every 8 connection(s)
I20250114 03:05:10.129256 5888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250114 03:05:10.130924 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 5888
I20250114 03:05:10.131037 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250114 03:05:10.133342 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.69:0
--local_ip_for_outbound_sockets=127.1.109.69
--webserver_interface=127.1.109.69
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250114 03:05:10.139195 6016 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:10.139441 6016 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:10.140035 6016 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:10.141202 5435 ts_manager.cc:194] Registered new tserver with Master: 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193)
I20250114 03:05:10.141795 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.68:33587
W20250114 03:05:10.243533 6019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:10.243775 6019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:10.243849 6019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:10.246884 6019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:10.247021 6019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.69
I20250114 03:05:10.250712 6019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.1.109.69
--webserver_port=0
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.69
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:10.251798 6019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:10.252915 6019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:10.261667 6024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:10.262589 6025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:10.263497 6027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:10.264389 6019 server_base.cc:1034] running on GCE node
I20250114 03:05:10.264740 6019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:10.265355 6019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:10.266510 6019 hybrid_clock.cc:648] HybridClock initialized: now 1736823910266493 us; error 34 us; skew 500 ppm
I20250114 03:05:10.271884 6019 webserver.cc:458] Webserver started at http://127.1.109.69:35807/ using document root <none> and password file <none>
I20250114 03:05:10.272481 6019 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:10.272541 6019 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:10.272744 6019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:10.274593 6019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7d03fcd3ed5c4594ad6f318777c3836f"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:10.275103 6019 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7d03fcd3ed5c4594ad6f318777c3836f"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:10.278707 6019 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:10.281222 6033 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:10.282313 6019 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:10.282428 6019 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7d03fcd3ed5c4594ad6f318777c3836f"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:10.282531 6019 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:10.298604 6019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:10.299278 6019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:10.299438 6019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:10.300004 6019 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:10.300931 6019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:10.300979 6019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:10.301023 6019 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:10.301054 6019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:10.310667 6019 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.69:46149
I20250114 03:05:10.310698 6146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.69:46149 every 8 connection(s)
I20250114 03:05:10.311902 6019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250114 03:05:10.320080 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6019
I20250114 03:05:10.320256 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250114 03:05:10.325246 6147 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:10.325539 6147 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:10.326241 6147 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:10.327380 5435 ts_manager.cc:194] Registered new tserver with Master: 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149)
I20250114 03:05:10.328133 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.69:47975
I20250114 03:05:10.335721 1461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250114 03:05:10.353710 5435 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34412:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250114 03:05:10.356570 5435 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20250114 03:05:10.375448 5950 tablet_service.cc:1467] Processing CreateTablet for tablet e50fa58b257940589757bfa9c8487032 (DEFAULT_TABLE table=TestTable [id=b8a15073b3714652a57a9c7eefcd0288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:10.376165 5557 tablet_service.cc:1467] Processing CreateTablet for tablet e50fa58b257940589757bfa9c8487032 (DEFAULT_TABLE table=TestTable [id=b8a15073b3714652a57a9c7eefcd0288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:10.376533 5950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50fa58b257940589757bfa9c8487032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:10.377142 5557 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50fa58b257940589757bfa9c8487032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:10.376791 6080 tablet_service.cc:1467] Processing CreateTablet for tablet e50fa58b257940589757bfa9c8487032 (DEFAULT_TABLE table=TestTable [id=b8a15073b3714652a57a9c7eefcd0288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:10.377962 6080 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50fa58b257940589757bfa9c8487032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:10.380146 5688 tablet_service.cc:1467] Processing CreateTablet for tablet e50fa58b257940589757bfa9c8487032 (DEFAULT_TABLE table=TestTable [id=b8a15073b3714652a57a9c7eefcd0288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:10.380789 5688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50fa58b257940589757bfa9c8487032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:10.384397 6169 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Bootstrap starting.
I20250114 03:05:10.384924 6170 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Bootstrap starting.
I20250114 03:05:10.387173 6168 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1: Bootstrap starting.
I20250114 03:05:10.387245 6170 tablet_bootstrap.cc:654] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:10.388075 6170 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:10.388514 5819 tablet_service.cc:1467] Processing CreateTablet for tablet e50fa58b257940589757bfa9c8487032 (DEFAULT_TABLE table=TestTable [id=b8a15073b3714652a57a9c7eefcd0288]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:10.389277 6168 tablet_bootstrap.cc:654] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:10.389544 5819 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e50fa58b257940589757bfa9c8487032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:10.389977 6168 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:10.391055 6169 tablet_bootstrap.cc:654] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:10.391811 6169 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:10.393184 6168 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1: No bootstrap required, opened a new log
I20250114 03:05:10.393352 6168 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250114 03:05:10.393610 6169 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: No bootstrap required, opened a new log
I20250114 03:05:10.393797 6169 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250114 03:05:10.395736 6171 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: Bootstrap starting.
I20250114 03:05:10.396958 6168 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.397212 6168 raft_consensus.cc:383] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:10.397267 6168 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ca84a15c67f41998b81c4543d8c67f1, State: Initialized, Role: FOLLOWER
I20250114 03:05:10.397500 6169 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.397709 6169 raft_consensus.cc:383] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:10.397770 6169 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d03fcd3ed5c4594ad6f318777c3836f, State: Initialized, Role: FOLLOWER
I20250114 03:05:10.397766 6168 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.398049 6171 tablet_bootstrap.cc:654] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:10.398212 6169 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.398535 6170 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: No bootstrap required, opened a new log
I20250114 03:05:10.398666 6168 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:10.398713 6170 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250114 03:05:10.398806 6171 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:10.399036 6016 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:10.399080 6169 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:10.399431 6147 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:10.399871 6175 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: Bootstrap starting.
I20250114 03:05:10.402860 6171 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: No bootstrap required, opened a new log
I20250114 03:05:10.403106 6171 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250114 03:05:10.403895 6170 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.404160 6170 raft_consensus.cc:383] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:10.404233 6170 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea06074428644ac86b5a52eab7554e6, State: Initialized, Role: FOLLOWER
I20250114 03:05:10.404693 6170 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.405570 5623 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:10.405740 6170 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250114 03:05:10.406821 6171 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.407033 6171 raft_consensus.cc:383] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:10.407095 6171 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44a39e7d733b422e8db32bae91ff0d56, State: Initialized, Role: FOLLOWER
I20250114 03:05:10.407536 6171 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.408272 5754 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:10.408495 6171 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:10.410172 6175 tablet_bootstrap.cc:654] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:10.410998 6175 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:10.412743 6175 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: No bootstrap required, opened a new log
I20250114 03:05:10.412956 6175 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20250114 03:05:10.415904 6175 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.416101 6175 raft_consensus.cc:383] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:10.416162 6175 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fc998e25ad148bb9d6245ba68a93b63, State: Initialized, Role: FOLLOWER
I20250114 03:05:10.416556 6175 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.417243 5885 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:10.417371 6175 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250114 03:05:10.451221 5886 tablet.cc:2367] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:10.497016 6179 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:05:10.497201 6179 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.498454 6179 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:10.503837 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56" is_pre_election: true
I20250114 03:05:10.503841 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1" is_pre_election: true
I20250114 03:05:10.504098 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63" is_pre_election: true
I20250114 03:05:10.504156 5708 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 0.
I20250114 03:05:10.504156 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 0.
I20250114 03:05:10.504402 5839 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 0.
I20250114 03:05:10.504559 6101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" is_pre_election: true
I20250114 03:05:10.504767 5510 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63, 9ea06074428644ac86b5a52eab7554e6; no voters:
I20250114 03:05:10.504801 6101 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 0.
I20250114 03:05:10.505066 6179 raft_consensus.cc:2798] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:05:10.505148 6179 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:05:10.505191 6179 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:10.506268 6179 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.506691 6179 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [CANDIDATE]: Term 1 election: Requested vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:10.507021 6101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d03fcd3ed5c4594ad6f318777c3836f"
I20250114 03:05:10.507021 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1"
I20250114 03:05:10.507036 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63"
I20250114 03:05:10.507007 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "9ea06074428644ac86b5a52eab7554e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56"
I20250114 03:05:10.507138 6101 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:10.507139 5970 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:10.507150 5839 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:10.507151 5708 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:10.508262 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 1.
I20250114 03:05:10.508261 6101 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 1.
I20250114 03:05:10.508265 5839 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 1.
I20250114 03:05:10.508265 5708 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ea06074428644ac86b5a52eab7554e6 in term 1.
I20250114 03:05:10.508611 5510 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 4 yes votes; 0 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63, 44a39e7d733b422e8db32bae91ff0d56, 9ea06074428644ac86b5a52eab7554e6; no voters:
I20250114 03:05:10.508790 6179 raft_consensus.cc:2798] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:10.509013 6179 raft_consensus.cc:695] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 1 LEADER]: Becoming Leader. State: Replica: 9ea06074428644ac86b5a52eab7554e6, State: Running, Role: LEADER
I20250114 03:05:10.509347 6179 consensus_queue.cc:237] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:10.512734 5434 catalog_manager.cc:5526] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "9ea06074428644ac86b5a52eab7554e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } health_report { overall_health: UNKNOWN } } }
W20250114 03:05:10.520682 5755 tablet.cc:2367] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:10.527449 1461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250114 03:05:10.529096 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9ea06074428644ac86b5a52eab7554e6 to finish bootstrapping
I20250114 03:05:10.534883 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 44a39e7d733b422e8db32bae91ff0d56 to finish bootstrapping
I20250114 03:05:10.540078 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1fc998e25ad148bb9d6245ba68a93b63 to finish bootstrapping
I20250114 03:05:10.545063 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0ca84a15c67f41998b81c4543d8c67f1 to finish bootstrapping
I20250114 03:05:10.549983 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7d03fcd3ed5c4594ad6f318777c3836f to finish bootstrapping
I20250114 03:05:10.559966 1461 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (5 alive)...
W20250114 03:05:10.563158 6148 tablet.cc:2367] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:10.573647 5708 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 1 FOLLOWER]: Refusing update from remote peer 9ea06074428644ac86b5a52eab7554e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:10.573647 6101 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Refusing update from remote peer 9ea06074428644ac86b5a52eab7554e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:10.573748 5839 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Refusing update from remote peer 9ea06074428644ac86b5a52eab7554e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:10.573748 5970 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Refusing update from remote peer 9ea06074428644ac86b5a52eab7554e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:10.574368 6193 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:10.574599 6187 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:10.574710 6179 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:10.575017 6194 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250114 03:05:10.580518 5624 tablet.cc:2367] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:10.587850 6192 mvcc.cc:204] Tried to move back new op lower bound from 7114030737702195200 to 7114030737449324544. Current Snapshot: MvccSnapshot[applied={T|T < 7114030737702195200}]
I20250114 03:05:10.588452 6195 mvcc.cc:204] Tried to move back new op lower bound from 7114030737702195200 to 7114030737449324544. Current Snapshot: MvccSnapshot[applied={T|T < 7114030737702195200}]
I20250114 03:05:10.589501 6198 mvcc.cc:204] Tried to move back new op lower bound from 7114030737702195200 to 7114030737449324544. Current Snapshot: MvccSnapshot[applied={T|T < 7114030737702195200}]
I20250114 03:05:10.593827 6196 mvcc.cc:204] Tried to move back new op lower bound from 7114030737702195200 to 7114030737449324544. Current Snapshot: MvccSnapshot[applied={T|T < 7114030737702195200}]
I20250114 03:05:10.597361 6197 mvcc.cc:204] Tried to move back new op lower bound from 7114030737702195200 to 7114030737449324544. Current Snapshot: MvccSnapshot[applied={T|T < 7114030737702195200}]
I20250114 03:05:10.616706 1461 raft_consensus_election-itest.cc:408] Killing current leader 9ea06074428644ac86b5a52eab7554e6...
I20250114 03:05:10.616852 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 5495
I20250114 03:05:10.623752 1461 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20250114 03:05:10.625581 1461 test_util.cc:274] Using random seed: -734334682
I20250114 03:05:10.744526 1461 test_util.cc:274] Using random seed: -734215739
I20250114 03:05:10.951223 1461 test_util.cc:274] Using random seed: -734009043
I20250114 03:05:11.258107 1461 test_util.cc:274] Using random seed: -733702158
I20250114 03:05:11.664480 1461 test_util.cc:274] Using random seed: -733295786
I20250114 03:05:12.116181 6214 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ea06074428644ac86b5a52eab7554e6)
I20250114 03:05:12.116271 6215 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ea06074428644ac86b5a52eab7554e6)
I20250114 03:05:12.116341 6216 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ea06074428644ac86b5a52eab7554e6)
I20250114 03:05:12.116355 6214 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:12.116405 6215 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:12.116484 6216 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:12.117024 6217 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 9ea06074428644ac86b5a52eab7554e6)
I20250114 03:05:12.117187 6217 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:12.118999 6215 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:12.119196 6214 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
W20250114 03:05:12.119436 5903 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.121199 6216 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579)
W20250114 03:05:12.121724 6034 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
W20250114 03:05:12.122128 5772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.122285 6217 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:12.124370 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1" is_pre_election: true
I20250114 03:05:12.124513 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d03fcd3ed5c4594ad6f318777c3836f in term 1.
W20250114 03:05:12.124657 5641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
W20250114 03:05:12.125696 5772 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.127028 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1" is_pre_election: true
I20250114 03:05:12.127151 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc998e25ad148bb9d6245ba68a93b63 in term 1.
W20250114 03:05:12.127341 5903 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.128656 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63" is_pre_election: true
I20250114 03:05:12.128845 5839 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d03fcd3ed5c4594ad6f318777c3836f in term 1.
I20250114 03:05:12.129431 6034 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63, 7d03fcd3ed5c4594ad6f318777c3836f; no voters:
I20250114 03:05:12.129726 6215 raft_consensus.cc:2798] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 03:05:12.129781 6215 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 9ea06074428644ac86b5a52eab7554e6)
I20250114 03:05:12.129829 6215 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:12.130560 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "0ca84a15c67f41998b81c4543d8c67f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63" is_pre_election: true
I20250114 03:05:12.130671 5839 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ca84a15c67f41998b81c4543d8c67f1 in term 1.
I20250114 03:05:12.130985 6215 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:12.131577 6215 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [CANDIDATE]: Term 2 election: Requested vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:12.132230 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1"
I20250114 03:05:12.132339 5970 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:12.132439 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56" is_pre_election: true
I20250114 03:05:12.132584 5708 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc998e25ad148bb9d6245ba68a93b63 in term 1.
I20250114 03:05:12.132927 5774 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63, 44a39e7d733b422e8db32bae91ff0d56; no voters: 9ea06074428644ac86b5a52eab7554e6
I20250114 03:05:12.133204 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63"
I20250114 03:05:12.133311 5839 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 1 FOLLOWER]: Advancing to term 2
W20250114 03:05:12.133517 6034 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.133728 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d03fcd3ed5c4594ad6f318777c3836f in term 2.
I20250114 03:05:12.134159 5707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56"
W20250114 03:05:12.134176 6034 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.134251 5707 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:12.134320 5839 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d03fcd3ed5c4594ad6f318777c3836f in term 2.
I20250114 03:05:12.135229 5707 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d03fcd3ed5c4594ad6f318777c3836f in term 2.
I20250114 03:05:12.135527 6036 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 1fc998e25ad148bb9d6245ba68a93b63, 44a39e7d733b422e8db32bae91ff0d56, 7d03fcd3ed5c4594ad6f318777c3836f; no voters: 9ea06074428644ac86b5a52eab7554e6
I20250114 03:05:12.135710 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56" is_pre_election: true
I20250114 03:05:12.135846 5708 raft_consensus.cc:2371] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7d03fcd3ed5c4594ad6f318777c3836f in term 2. Re-sending same reply.
I20250114 03:05:12.136687 6215 raft_consensus.cc:2798] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:05:12.136916 6215 raft_consensus.cc:695] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 2 LEADER]: Becoming Leader. State: Replica: 7d03fcd3ed5c4594ad6f318777c3836f, State: Running, Role: LEADER
I20250114 03:05:12.137127 6100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "0ca84a15c67f41998b81c4543d8c67f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" is_pre_election: true
I20250114 03:05:12.137246 6215 consensus_queue.cc:237] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
W20250114 03:05:12.137838 5641 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:12.140125 6216 raft_consensus.cc:2758] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20250114 03:05:12.140084 5432 catalog_manager.cc:5526] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f reported cstate change: term changed from 1 to 2, leader changed from 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65) to 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69). New cstate: current_term: 2 leader_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } health_report { overall_health: UNKNOWN } } }
I20250114 03:05:12.141196 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "44a39e7d733b422e8db32bae91ff0d56" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63" is_pre_election: true
I20250114 03:05:12.141343 5839 raft_consensus.cc:2388] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44a39e7d733b422e8db32bae91ff0d56 in current term 2: Already voted for candidate 7d03fcd3ed5c4594ad6f318777c3836f in this term.
I20250114 03:05:12.141366 6100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "44a39e7d733b422e8db32bae91ff0d56" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" is_pre_election: true
I20250114 03:05:12.141360 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "0ca84a15c67f41998b81c4543d8c67f1" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56" is_pre_election: true
I20250114 03:05:12.141491 5708 raft_consensus.cc:2388] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ca84a15c67f41998b81c4543d8c67f1 in current term 2: Already voted for candidate 7d03fcd3ed5c4594ad6f318777c3836f in this term.
I20250114 03:05:12.141850 5642 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 44a39e7d733b422e8db32bae91ff0d56; no voters: 1fc998e25ad148bb9d6245ba68a93b63, 7d03fcd3ed5c4594ad6f318777c3836f, 9ea06074428644ac86b5a52eab7554e6
I20250114 03:05:12.141886 5905 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63; no voters: 44a39e7d733b422e8db32bae91ff0d56, 7d03fcd3ed5c4594ad6f318777c3836f, 9ea06074428644ac86b5a52eab7554e6
I20250114 03:05:12.142005 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "44a39e7d733b422e8db32bae91ff0d56" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1" is_pre_election: true
I20250114 03:05:12.142165 5970 raft_consensus.cc:2388] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 44a39e7d733b422e8db32bae91ff0d56 in current term 2: Already voted for candidate 7d03fcd3ed5c4594ad6f318777c3836f in this term.
I20250114 03:05:12.142158 6217 raft_consensus.cc:2743] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:05:12.142290 6214 raft_consensus.cc:2743] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250114 03:05:12.143451 6100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" is_pre_election: true
I20250114 03:05:12.171809 1461 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (4 alive)...
W20250114 03:05:12.174567 6150 meta_cache.cc:302] tablet e50fa58b257940589757bfa9c8487032: replica 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579) has failed: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
W20250114 03:05:12.184008 5668 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33920: Illegal state: replica 44a39e7d733b422e8db32bae91ff0d56 is not leader of this config: current role FOLLOWER
W20250114 03:05:12.191702 5930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39258: Illegal state: replica 0ca84a15c67f41998b81c4543d8c67f1 is not leader of this config: current role FOLLOWER
I20250114 03:05:12.204962 5839 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Refusing update from remote peer 7d03fcd3ed5c4594ad6f318777c3836f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250114 03:05:12.205021 5970 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Refusing update from remote peer 7d03fcd3ed5c4594ad6f318777c3836f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250114 03:05:12.205060 5708 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Refusing update from remote peer 7d03fcd3ed5c4594ad6f318777c3836f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20250114 03:05:12.205569 6250 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20250114 03:05:12.205754 6034 consensus_peers.cc:487] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f -> Peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Couldn't send request to peer 9ea06074428644ac86b5a52eab7554e6. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:05:12.205787 6260 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 03:05:12.206111 6260 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 03:05:12.234776 1461 raft_consensus_election-itest.cc:408] Killing current leader 7d03fcd3ed5c4594ad6f318777c3836f...
I20250114 03:05:12.234901 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6019
I20250114 03:05:12.242617 1461 raft_consensus_election-itest.cc:412] Waiting for new guy to be elected leader.
I20250114 03:05:12.244513 1461 test_util.cc:274] Using random seed: -732715750
I20250114 03:05:12.350317 1461 test_util.cc:274] Using random seed: -732609948
I20250114 03:05:12.556627 1461 test_util.cc:274] Using random seed: -732403639
I20250114 03:05:12.863063 1461 test_util.cc:274] Using random seed: -732097202
I20250114 03:05:13.269415 1461 test_util.cc:274] Using random seed: -731690850
I20250114 03:05:13.734874 6273 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d03fcd3ed5c4594ad6f318777c3836f)
I20250114 03:05:13.734891 6274 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d03fcd3ed5c4594ad6f318777c3836f)
I20250114 03:05:13.735013 6273 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:13.735018 6274 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:13.735460 6275 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7d03fcd3ed5c4594ad6f318777c3836f)
I20250114 03:05:13.735600 6273 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579)
I20250114 03:05:13.735977 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56" is_pre_election: true
I20250114 03:05:13.736114 5708 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc998e25ad148bb9d6245ba68a93b63 in term 2.
I20250114 03:05:13.736343 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1" is_pre_election: true
I20250114 03:05:13.736373 5969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "44a39e7d733b422e8db32bae91ff0d56" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1" is_pre_election: true
I20250114 03:05:13.736891 6275 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:13.737126 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fc998e25ad148bb9d6245ba68a93b63 in term 2.
I20250114 03:05:13.737309 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "44a39e7d733b422e8db32bae91ff0d56" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63" is_pre_election: true
I20250114 03:05:13.737425 5839 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44a39e7d733b422e8db32bae91ff0d56 in term 2.
I20250114 03:05:13.737463 5773 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63, 44a39e7d733b422e8db32bae91ff0d56; no voters:
W20250114 03:05:13.737545 5642 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149): Network error: Client connection negotiation failed: client connection to 127.1.109.69:46149: connect: Connection refused (error 111)
I20250114 03:05:13.737613 6273 raft_consensus.cc:2798] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 03:05:13.737668 6273 raft_consensus.cc:491] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7d03fcd3ed5c4594ad6f318777c3836f)
I20250114 03:05:13.737708 6273 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:05:13.738054 6274 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
W20250114 03:05:13.738163 5772 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
W20250114 03:05:13.738322 5773 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149): Network error: Client connection negotiation failed: client connection to 127.1.109.69:46149: connect: Connection refused (error 111)
W20250114 03:05:13.738353 5641 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:13.738413 5641 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 1fc998e25ad148bb9d6245ba68a93b63, 44a39e7d733b422e8db32bae91ff0d56; no voters: 0ca84a15c67f41998b81c4543d8c67f1, 7d03fcd3ed5c4594ad6f318777c3836f, 9ea06074428644ac86b5a52eab7554e6
I20250114 03:05:13.738706 6275 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579), 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577)
I20250114 03:05:13.738920 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "0ca84a15c67f41998b81c4543d8c67f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56" is_pre_election: true
I20250114 03:05:13.739032 5708 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ca84a15c67f41998b81c4543d8c67f1 in term 2.
I20250114 03:05:13.739084 6274 raft_consensus.cc:2743] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 03:05:13.739596 5839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "0ca84a15c67f41998b81c4543d8c67f1" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "1fc998e25ad148bb9d6245ba68a93b63" is_pre_election: true
W20250114 03:05:13.739825 5904 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149): Network error: Client connection negotiation failed: client connection to 127.1.109.69:46149: connect: Connection refused (error 111)
W20250114 03:05:13.740058 5903 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:13.740332 6273 raft_consensus.cc:513] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:13.740525 5839 raft_consensus.cc:2388] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0ca84a15c67f41998b81c4543d8c67f1 in current term 3: Already voted for candidate 1fc998e25ad148bb9d6245ba68a93b63 in this term.
I20250114 03:05:13.740808 6273 leader_election.cc:290] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 election: Requested vote from peers 44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653), 0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193), 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149), 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579)
I20250114 03:05:13.741078 5970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "0ca84a15c67f41998b81c4543d8c67f1"
I20250114 03:05:13.741097 5708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50fa58b257940589757bfa9c8487032" candidate_uuid: "1fc998e25ad148bb9d6245ba68a93b63" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "44a39e7d733b422e8db32bae91ff0d56"
I20250114 03:05:13.741185 5970 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:05:13.741199 5708 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:05:13.741331 5903 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 44a39e7d733b422e8db32bae91ff0d56; no voters: 1fc998e25ad148bb9d6245ba68a93b63, 7d03fcd3ed5c4594ad6f318777c3836f, 9ea06074428644ac86b5a52eab7554e6
W20250114 03:05:13.742014 5773 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149): Network error: Client connection negotiation failed: client connection to 127.1.109.69:46149: connect: Connection refused (error 111)
W20250114 03:05:13.742094 5772 leader_election.cc:336] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111)
I20250114 03:05:13.742208 5970 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc998e25ad148bb9d6245ba68a93b63 in term 3.
I20250114 03:05:13.742226 5708 raft_consensus.cc:2463] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fc998e25ad148bb9d6245ba68a93b63 in term 3.
I20250114 03:05:13.742317 6275 raft_consensus.cc:2743] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250114 03:05:13.742516 5773 leader_election.cc:304] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 0ca84a15c67f41998b81c4543d8c67f1, 1fc998e25ad148bb9d6245ba68a93b63, 44a39e7d733b422e8db32bae91ff0d56; no voters: 7d03fcd3ed5c4594ad6f318777c3836f, 9ea06074428644ac86b5a52eab7554e6
I20250114 03:05:13.742668 6273 raft_consensus.cc:2798] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:05:13.742849 6273 raft_consensus.cc:695] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [term 3 LEADER]: Becoming Leader. State: Replica: 1fc998e25ad148bb9d6245ba68a93b63, State: Running, Role: LEADER
I20250114 03:05:13.743166 6273 consensus_queue.cc:237] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:13.746019 5432 catalog_manager.cc:5526] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 reported cstate change: term changed from 2 to 3, leader changed from 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69) to 1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67). New cstate: current_term: 3 leader_uuid: "1fc998e25ad148bb9d6245ba68a93b63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:13.775609 1461 raft_consensus_election-itest.cc:397] Writing data to leader of 5-node config (3 alive)...
W20250114 03:05:13.778167 6151 meta_cache.cc:302] tablet e50fa58b257940589757bfa9c8487032: replica 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149) has failed: Network error: Client connection negotiation failed: client connection to 127.1.109.69:46149: connect: Connection refused (error 111)
W20250114 03:05:13.780285 5668 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:33920: Illegal state: replica 44a39e7d733b422e8db32bae91ff0d56 is not leader of this config: current role FOLLOWER
W20250114 03:05:13.784715 5930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:39258: Illegal state: replica 0ca84a15c67f41998b81c4543d8c67f1 is not leader of this config: current role FOLLOWER
I20250114 03:05:13.796142 5708 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 44a39e7d733b422e8db32bae91ff0d56 [term 3 FOLLOWER]: Refusing update from remote peer 1fc998e25ad148bb9d6245ba68a93b63: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250114 03:05:13.796342 5970 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 0ca84a15c67f41998b81c4543d8c67f1 [term 3 FOLLOWER]: Refusing update from remote peer 1fc998e25ad148bb9d6245ba68a93b63: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250114 03:05:13.796680 6296 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250114 03:05:13.796905 5772 consensus_peers.cc:487] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 -> Peer 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): Couldn't send request to peer 9ea06074428644ac86b5a52eab7554e6. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.65:42579: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:05:13.796921 6286 consensus_queue.cc:1035] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250114 03:05:13.797363 5773 consensus_peers.cc:487] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 -> Peer 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149): Couldn't send request to peer 7d03fcd3ed5c4594ad6f318777c3836f. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.69:46149: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:05:13.820230 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.69:46149
--local_ip_for_outbound_sockets=127.1.109.69
--tserver_master_addrs=127.1.109.126:42231
--webserver_port=35807
--webserver_interface=127.1.109.69
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20250114 03:05:13.927852 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:13.928068 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:13.928117 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:13.931324 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:13.931433 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.69
I20250114 03:05:13.935184 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.69:46149
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.1.109.69
--webserver_port=35807
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.69
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:13.936278 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:13.937516 6306 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:13.945885 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:13.946483 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:13.947422 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:13.948282 6306 server_base.cc:1034] running on GCE node
I20250114 03:05:13.948668 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:13.949270 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:13.950455 6306 hybrid_clock.cc:648] HybridClock initialized: now 1736823913950428 us; error 43 us; skew 500 ppm
I20250114 03:05:13.958120 6306 webserver.cc:458] Webserver started at http://127.1.109.69:35807/ using document root <none> and password file <none>
I20250114 03:05:13.958889 6306 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:13.958978 6306 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:13.962845 6306 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20250114 03:05:13.964772 6320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:13.965795 6306 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:13.965895 6306 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7d03fcd3ed5c4594ad6f318777c3836f"
format_stamp: "Formatted at 2025-01-14 03:05:10 on dist-test-slave-r16g"
I20250114 03:05:13.966320 6306 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:13.985605 6306 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:13.986311 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:13.986481 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:13.987078 6306 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:13.988224 6327 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 03:05:13.991613 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:05:13.991672 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250114 03:05:13.991729 6306 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 03:05:13.993284 6306 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:05:13.993327 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:13.993594 6327 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Bootstrap starting.
I20250114 03:05:14.003531 6306 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.69:46149
I20250114 03:05:14.003576 6434 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.69:46149 every 8 connection(s)
I20250114 03:05:14.004570 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250114 03:05:14.006338 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6306
I20250114 03:05:14.006822 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:42579
--local_ip_for_outbound_sockets=127.1.109.65
--tserver_master_addrs=127.1.109.126:42231
--webserver_port=36385
--webserver_interface=127.1.109.65
--builtin_ntp_servers=127.1.109.84:42271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
I20250114 03:05:14.009750 6327 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:14.014153 6435 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:14.014412 6435 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:14.014956 6435 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:14.016657 5427 ts_manager.cc:194] Re-registered known tserver with Master: 7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149)
I20250114 03:05:14.017475 5427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.69:56077
I20250114 03:05:14.027046 6327 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=100 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:05:14.027531 6327 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Bootstrap complete.
I20250114 03:05:14.028873 6327 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.008s
I20250114 03:05:14.031487 6327 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:14.031903 6327 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d03fcd3ed5c4594ad6f318777c3836f, State: Initialized, Role: FOLLOWER
I20250114 03:05:14.032370 6327 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:14.033105 6435 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:14.033289 6327 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250114 03:05:14.122020 6439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:14.122238 6439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:14.122287 6439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:14.125449 6439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:14.125679 6439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:14.129812 6439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:42271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:42579
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=36385
--tserver_master_addrs=127.1.109.126:42231
--never_fsync=true
--heap_profile_path=/tmp/kudu.6439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:14.130990 6439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:14.132171 6439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:14.141343 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:14.142256 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:14.143106 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:14.144070 6439 server_base.cc:1034] running on GCE node
I20250114 03:05:14.144470 6439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:14.145108 6439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:14.146266 6439 hybrid_clock.cc:648] HybridClock initialized: now 1736823914146255 us; error 26 us; skew 500 ppm
I20250114 03:05:14.148252 6439 webserver.cc:458] Webserver started at http://127.1.109.65:36385/ using document root <none> and password file <none>
I20250114 03:05:14.148871 6439 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:14.148952 6439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:14.152366 6439 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:14.154325 6455 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:14.155360 6439 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:14.155469 6439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9ea06074428644ac86b5a52eab7554e6"
format_stamp: "Formatted at 2025-01-14 03:05:09 on dist-test-slave-r16g"
I20250114 03:05:14.155901 6439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:14.177026 6439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:14.177740 6439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:14.177914 6439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:14.178476 6439 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:14.179695 6462 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 03:05:14.182950 6439 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:05:14.183013 6439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250114 03:05:14.183086 6439 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 03:05:14.184577 6439 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:05:14.184624 6439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250114 03:05:14.184916 6462 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Bootstrap starting.
I20250114 03:05:14.194995 6439 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:42579
I20250114 03:05:14.195027 6569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:42579 every 8 connection(s)
I20250114 03:05:14.196070 6439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:14.200675 6462 log.cc:826] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:14.203380 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6439
I20250114 03:05:14.205488 6570 heartbeater.cc:346] Connected to a master server at 127.1.109.126:42231
I20250114 03:05:14.205791 6570 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:14.206306 6570 heartbeater.cc:510] Master 127.1.109.126:42231 requested a full tablet report, sending...
I20250114 03:05:14.208025 5427 ts_manager.cc:194] Re-registered known tserver with Master: 9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579)
I20250114 03:05:14.208992 5427 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:38825
I20250114 03:05:14.211565 6462 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=50 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:05:14.211975 6462 tablet_bootstrap.cc:492] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Bootstrap complete.
I20250114 03:05:14.213227 6462 ts_tablet_manager.cc:1397] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.010s
I20250114 03:05:14.215517 6462 raft_consensus.cc:357] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:14.215867 6462 raft_consensus.cc:738] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ea06074428644ac86b5a52eab7554e6, State: Initialized, Role: FOLLOWER
I20250114 03:05:14.216297 6462 consensus_queue.cc:260] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44a39e7d733b422e8db32bae91ff0d56" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 34653 } } peers { permanent_uuid: "0ca84a15c67f41998b81c4543d8c67f1" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 34193 } } peers { permanent_uuid: "7d03fcd3ed5c4594ad6f318777c3836f" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 46149 } } peers { permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 } } peers { permanent_uuid: "1fc998e25ad148bb9d6245ba68a93b63" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 38577 } }
I20250114 03:05:14.216984 6570 heartbeater.cc:502] Master 127.1.109.126:42231 was elected leader, sending a full tablet report...
I20250114 03:05:14.217180 6462 ts_tablet_manager.cc:1428] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250114 03:05:14.253507 6389 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 7d03fcd3ed5c4594ad6f318777c3836f [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:05:14.259670 5819 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:05:14.263523 6604 mvcc.cc:204] Tried to move back new op lower bound from 7114030750995189760 to 7114030750695792640. Current Snapshot: MvccSnapshot[applied={T|T < 7114030750903861248}]
I20250114 03:05:14.267123 5688 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:14.273886 6503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:14.274317 5950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:14.282650 6368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
0316a506daeb475f957ba2e84cee9332 | 127.1.109.126:42231 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:42271 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0ca84a15c67f41998b81c4543d8c67f1 | 127.1.109.68:34193 | HEALTHY | <none> | 0 | 0
1fc998e25ad148bb9d6245ba68a93b63 | 127.1.109.67:38577 | HEALTHY | <none> | 1 | 0
44a39e7d733b422e8db32bae91ff0d56 | 127.1.109.66:34653 | HEALTHY | <none> | 0 | 0
7d03fcd3ed5c4594ad6f318777c3836f | 127.1.109.69:46149 | HEALTHY | <none> | 0 | 0
9ea06074428644ac86b5a52eab7554e6 | 127.1.109.65:42579 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:42579
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:34653
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:38577
local_ip_for_outbound_sockets | 127.1.109.68 | experimental | 127.1.109.68:34193
local_ip_for_outbound_sockets | 127.1.109.69 | experimental | 127.1.109.69:46149
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:42579
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:34653
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:38577
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.1.109.68:34193
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.1.109.69:46149
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:42271 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
Tablet e50fa58b257940589757bfa9c8487032 of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
44a39e7d733b422e8db32bae91ff0d56 (127.1.109.66:34653): RUNNING
0ca84a15c67f41998b81c4543d8c67f1 (127.1.109.68:34193): RUNNING
7d03fcd3ed5c4594ad6f318777c3836f (127.1.109.69:46149): RUNNING
9ea06074428644ac86b5a52eab7554e6 (127.1.109.65:42579): RUNNING
1fc998e25ad148bb9d6245ba68a93b63 (127.1.109.67:38577): RUNNING [LEADER]
All reported replicas are:
A = 44a39e7d733b422e8db32bae91ff0d56
B = 0ca84a15c67f41998b81c4543d8c67f1
C = 7d03fcd3ed5c4594ad6f318777c3836f
D = 9ea06074428644ac86b5a52eab7554e6
E = 1fc998e25ad148bb9d6245ba68a93b63
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+----------------------+--------------+--------------+------------
master | A B C D E* | | | Yes
A | A B C D E* | 3 | -1 | Yes
B | A B C D E* | 3 | -1 | Yes
C | A B C D E* | 3 | -1 | Yes
D | A B C D E | 1 | -1 | Yes
E | A B C D E* | 3 | -1 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 5 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 1 table(s) are not healthy
FAILED
I20250114 03:05:14.306825 1461 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250114 03:05:14.404708 6524 raft_consensus.cc:3054] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 1 FOLLOWER]: Advancing to term 3
I20250114 03:05:14.406844 6524 raft_consensus.cc:1270] T e50fa58b257940589757bfa9c8487032 P 9ea06074428644ac86b5a52eab7554e6 [term 3 FOLLOWER]: Refusing update from remote peer 1fc998e25ad148bb9d6245ba68a93b63: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250114 03:05:14.407503 6297 consensus_queue.cc:1037] T e50fa58b257940589757bfa9c8487032 P 1fc998e25ad148bb9d6245ba68a93b63 [LEADER]: Got LMP mismatch error from peer: Peer: permanent_uuid: "9ea06074428644ac86b5a52eab7554e6" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 6, Time since last communication: 0.000s
I20250114 03:05:14.411916 6610 mvcc.cc:204] Tried to move back new op lower bound from 7114030744454135808 to 7114030744118677504. Current Snapshot: MvccSnapshot[applied={T|T < 7114030744385720320}]
I20250114 03:05:14.506657 5950 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:14.507421 5688 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:14.509078 6368 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20250114 03:05:14.510478 5819 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20250114 03:05:14.520589 6503 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
0316a506daeb475f957ba2e84cee9332 | 127.1.109.126:42231 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:42271 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0ca84a15c67f41998b81c4543d8c67f1 | 127.1.109.68:34193 | HEALTHY | <none> | 0 | 0
1fc998e25ad148bb9d6245ba68a93b63 | 127.1.109.67:38577 | HEALTHY | <none> | 1 | 0
44a39e7d733b422e8db32bae91ff0d56 | 127.1.109.66:34653 | HEALTHY | <none> | 0 | 0
7d03fcd3ed5c4594ad6f318777c3836f | 127.1.109.69:46149 | HEALTHY | <none> | 0 | 0
9ea06074428644ac86b5a52eab7554e6 | 127.1.109.65:42579 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 5
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 5 server(s) checked
local_ip_for_outbound_sockets | 127.1.109.65 | experimental | 127.1.109.65:42579
local_ip_for_outbound_sockets | 127.1.109.66 | experimental | 127.1.109.66:34653
local_ip_for_outbound_sockets | 127.1.109.67 | experimental | 127.1.109.67:38577
local_ip_for_outbound_sockets | 127.1.109.68 | experimental | 127.1.109.68:34193
local_ip_for_outbound_sockets | 127.1.109.69 | experimental | 127.1.109.69:46149
never_fsync | true | unsafe,advanced | all 5 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 5 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 5 server(s) checked
server_dump_info_format | pb | hidden | all 5 server(s) checked
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.1.109.65:42579
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.1.109.66:34653
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.1.109.67:38577
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.1.109.68:34193
server_dump_info_path | /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElection.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.1.109.69:46149
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.1.109.84:42271 | all 5 server(s) checked
time_source | builtin | all 5 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 6 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-----------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 5
Tables | 1
Tablets | 1
Replicas | 5
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250114 03:05:14.543296 1461 log_verifier.cc:126] Checking tablet e50fa58b257940589757bfa9c8487032
I20250114 03:05:14.588099 1461 log_verifier.cc:177] Verified matching terms for 18 ops in tablet e50fa58b257940589757bfa9c8487032
I20250114 03:05:14.606139 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6439
I20250114 03:05:14.612403 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 5626
I20250114 03:05:14.620625 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 5757
I20250114 03:05:14.626956 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 5888
I20250114 03:05:14.632680 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6306
I20250114 03:05:14.638685 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 5404
2025-01-14T03:05:14Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElection (5503 ms)
[ RUN ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica
I20250114 03:05:14.665194 1461 test_util.cc:274] Using random seed: -730295068
I20250114 03:05:14.665980 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:14.666025 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:14.666046 1461 ts_itest-base.cc:114] 1 tablet servers
I20250114 03:05:14.666064 1461 ts_itest-base.cc:115] 1 replicas per TS
I20250114 03:05:14.666088 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:14Z Disabled control of system clock
I20250114 03:05:14.674369 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:39897
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:39897 with env {}
W20250114 03:05:14.780431 6656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:14.780714 6656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:14.780787 6656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:14.783883 6656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:14.783946 6656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:14.783978 6656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:14.784008 6656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:14.787798 6656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:39897
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:39897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:14.788965 6656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:14.790163 6656 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:14.798033 6664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:14.798924 6661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:14.799731 6662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:14.800655 6656 server_base.cc:1034] running on GCE node
I20250114 03:05:14.801250 6656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:14.802222 6656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:14.803368 6656 hybrid_clock.cc:648] HybridClock initialized: now 1736823914803357 us; error 26 us; skew 500 ppm
I20250114 03:05:14.805382 6656 webserver.cc:458] Webserver started at http://127.1.109.126:41481/ using document root <none> and password file <none>
I20250114 03:05:14.805959 6656 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:14.806015 6656 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:14.806186 6656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:14.807957 6656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a78c456dc67f4db395ab45a9282f8507"
format_stamp: "Formatted at 2025-01-14 03:05:14 on dist-test-slave-r16g"
I20250114 03:05:14.808537 6656 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a78c456dc67f4db395ab45a9282f8507"
format_stamp: "Formatted at 2025-01-14 03:05:14 on dist-test-slave-r16g"
I20250114 03:05:14.812062 6656 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:05:14.814325 6670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:14.815364 6656 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:14.815474 6656 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a78c456dc67f4db395ab45a9282f8507"
format_stamp: "Formatted at 2025-01-14 03:05:14 on dist-test-slave-r16g"
I20250114 03:05:14.815624 6656 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:14.831266 6656 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:14.831895 6656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:14.832027 6656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:14.838429 6656 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:39897
I20250114 03:05:14.838447 6722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:39897 every 8 connection(s)
I20250114 03:05:14.839447 6656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:14.842453 6723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:14.847079 6723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507: Bootstrap starting.
I20250114 03:05:14.849385 6723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:14.849764 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6656
I20250114 03:05:14.849882 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:14.850270 6723 log.cc:826] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:14.852438 6723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507: No bootstrap required, opened a new log
I20250114 03:05:14.855954 6723 raft_consensus.cc:357] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78c456dc67f4db395ab45a9282f8507" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 39897 } }
I20250114 03:05:14.856195 6723 raft_consensus.cc:383] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:14.856310 6723 raft_consensus.cc:738] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78c456dc67f4db395ab45a9282f8507, State: Initialized, Role: FOLLOWER
I20250114 03:05:14.856760 6723 consensus_queue.cc:260] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78c456dc67f4db395ab45a9282f8507" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 39897 } }
I20250114 03:05:14.856925 6723 raft_consensus.cc:397] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:14.856998 6723 raft_consensus.cc:491] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:14.857128 6723 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:14.858124 6723 raft_consensus.cc:513] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78c456dc67f4db395ab45a9282f8507" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 39897 } }
I20250114 03:05:14.858467 6723 leader_election.cc:304] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a78c456dc67f4db395ab45a9282f8507; no voters:
I20250114 03:05:14.858747 6723 leader_election.cc:290] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:14.858834 6728 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:14.859040 6728 raft_consensus.cc:695] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [term 1 LEADER]: Becoming Leader. State: Replica: a78c456dc67f4db395ab45a9282f8507, State: Running, Role: LEADER
I20250114 03:05:14.859366 6728 consensus_queue.cc:237] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78c456dc67f4db395ab45a9282f8507" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 39897 } }
I20250114 03:05:14.859988 6723 sys_catalog.cc:564] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:14.860623 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a78c456dc67f4db395ab45a9282f8507. Latest consensus state: current_term: 1 leader_uuid: "a78c456dc67f4db395ab45a9282f8507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78c456dc67f4db395ab45a9282f8507" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 39897 } } }
I20250114 03:05:14.860862 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:14.861004 6729 sys_catalog.cc:455] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a78c456dc67f4db395ab45a9282f8507" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a78c456dc67f4db395ab45a9282f8507" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 39897 } } }
I20250114 03:05:14.861171 6729 sys_catalog.cc:458] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:14.861307 6736 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:14.864244 6736 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:14.869148 6736 catalog_manager.cc:1348] Generated new cluster ID: 4f8c410902ea45e99a98bdebf809301c
I20250114 03:05:14.869215 6736 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:14.902148 6736 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:14.903189 6736 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:14.908280 6736 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a78c456dc67f4db395ab45a9282f8507: Generated new TSK 0
I20250114 03:05:14.909025 6736 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:14.915112 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:39897
--builtin_ntp_servers=127.1.109.84:33637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=50 with env {}
W20250114 03:05:15.039829 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:15.040060 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:15.040135 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:15.043462 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:15.043617 6747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:15.047407 6747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=50
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:39897
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:15.048544 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:15.049738 6747 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:15.058442 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.059087 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.060001 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:15.060880 6747 server_base.cc:1034] running on GCE node
I20250114 03:05:15.061266 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:15.061949 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:15.063108 6747 hybrid_clock.cc:648] HybridClock initialized: now 1736823915063085 us; error 34 us; skew 500 ppm
I20250114 03:05:15.065438 6747 webserver.cc:458] Webserver started at http://127.1.109.65:37303/ using document root <none> and password file <none>
I20250114 03:05:15.066140 6747 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:15.066208 6747 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:15.066387 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:15.068123 6747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ea1973e6c3094c389a39d51cc875ad85"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.068640 6747 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ea1973e6c3094c389a39d51cc875ad85"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.072183 6747 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:15.074589 6761 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.075668 6747 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250114 03:05:15.075793 6747 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ea1973e6c3094c389a39d51cc875ad85"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.075886 6747 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:15.099146 6747 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:15.099798 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:15.099968 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:15.100571 6747 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:15.101526 6747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:15.101583 6747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.101651 6747 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:15.101677 6747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.110073 6747 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:40605
I20250114 03:05:15.110100 6874 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:40605 every 8 connection(s)
I20250114 03:05:15.111016 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:15.111894 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6747
I20250114 03:05:15.112001 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:15.120054 6875 heartbeater.cc:346] Connected to a master server at 127.1.109.126:39897
I20250114 03:05:15.120270 6875 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:15.120882 6875 heartbeater.cc:510] Master 127.1.109.126:39897 requested a full tablet report, sending...
I20250114 03:05:15.122364 6687 ts_manager.cc:194] Registered new tserver with Master: ea1973e6c3094c389a39d51cc875ad85 (127.1.109.65:40605)
I20250114 03:05:15.123543 6687 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:47365
I20250114 03:05:15.126127 1461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250114 03:05:15.134853 6687 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45342:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250114 03:05:15.151160 6809 tablet_service.cc:1467] Processing CreateTablet for tablet 5d18b746675f4cea941a198f50203154 (DEFAULT_TABLE table=TestTable [id=73623c7e3b584abdbcc70535f1c2b0db]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:15.152099 6809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d18b746675f4cea941a198f50203154. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:15.157300 6885 tablet_bootstrap.cc:492] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85: Bootstrap starting.
I20250114 03:05:15.159212 6885 tablet_bootstrap.cc:654] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:15.159888 6885 log.cc:826] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:15.161665 6885 tablet_bootstrap.cc:492] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85: No bootstrap required, opened a new log
I20250114 03:05:15.161849 6885 ts_tablet_manager.cc:1397] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:05:15.164685 6885 raft_consensus.cc:357] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1973e6c3094c389a39d51cc875ad85" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40605 } }
I20250114 03:05:15.164914 6885 raft_consensus.cc:383] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:15.164980 6885 raft_consensus.cc:738] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1973e6c3094c389a39d51cc875ad85, State: Initialized, Role: FOLLOWER
I20250114 03:05:15.165407 6885 consensus_queue.cc:260] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1973e6c3094c389a39d51cc875ad85" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40605 } }
I20250114 03:05:15.165534 6885 raft_consensus.cc:397] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:15.165601 6885 raft_consensus.cc:491] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:15.165712 6885 raft_consensus.cc:3054] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:15.166616 6885 raft_consensus.cc:513] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1973e6c3094c389a39d51cc875ad85" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40605 } }
I20250114 03:05:15.166962 6885 leader_election.cc:304] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea1973e6c3094c389a39d51cc875ad85; no voters:
I20250114 03:05:15.167304 6885 leader_election.cc:290] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:15.167446 6887 raft_consensus.cc:2798] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:15.167672 6887 raft_consensus.cc:695] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [term 1 LEADER]: Becoming Leader. State: Replica: ea1973e6c3094c389a39d51cc875ad85, State: Running, Role: LEADER
I20250114 03:05:15.167913 6875 heartbeater.cc:502] Master 127.1.109.126:39897 was elected leader, sending a full tablet report...
I20250114 03:05:15.168038 6887 consensus_queue.cc:237] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1973e6c3094c389a39d51cc875ad85" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40605 } }
I20250114 03:05:15.168172 6885 ts_tablet_manager.cc:1428] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250114 03:05:15.171908 6687 catalog_manager.cc:5526] T 5d18b746675f4cea941a198f50203154 P ea1973e6c3094c389a39d51cc875ad85 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea1973e6c3094c389a39d51cc875ad85 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "ea1973e6c3094c389a39d51cc875ad85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea1973e6c3094c389a39d51cc875ad85" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40605 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:15.177722 1461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250114 03:05:15.179083 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver ea1973e6c3094c389a39d51cc875ad85 to finish bootstrapping
I20250114 03:05:15.188951 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6747
I20250114 03:05:15.195932 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6656
2025-01-14T03:05:15Z chronyd exiting
[ OK ] RaftConsensusElectionITest.AutomaticLeaderElectionOneReplica (551 ms)
[ RUN ] RaftConsensusElectionITest.LeaderStepDown
I20250114 03:05:15.216506 1461 test_util.cc:274] Using random seed: -729743758
I20250114 03:05:15.217489 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:15.217541 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:15.217559 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:05:15.217576 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:05:15.217603 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:15Z Disabled control of system clock
I20250114 03:05:15.225486 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:44177
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:44177
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250114 03:05:15.332331 6900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:15.332600 6900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:15.332671 6900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:15.335858 6900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:15.335927 6900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:15.335961 6900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:15.335994 6900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:15.339834 6900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:44177
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:44177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.6900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:15.341042 6900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:15.342213 6900 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:15.349967 6906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.350525 6905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.351397 6908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:15.352216 6900 server_base.cc:1034] running on GCE node
I20250114 03:05:15.352751 6900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:15.353770 6900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:15.354959 6900 hybrid_clock.cc:648] HybridClock initialized: now 1736823915354945 us; error 28 us; skew 500 ppm
I20250114 03:05:15.357182 6900 webserver.cc:458] Webserver started at http://127.1.109.126:39687/ using document root <none> and password file <none>
I20250114 03:05:15.357774 6900 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:15.357837 6900 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:15.358002 6900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:15.359694 6900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "332ae850ae8e4607b86100b539404962"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.360165 6900 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "332ae850ae8e4607b86100b539404962"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.363426 6900 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:15.365600 6914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.366605 6900 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:15.366714 6900 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "332ae850ae8e4607b86100b539404962"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.366796 6900 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:15.377300 6900 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:15.377933 6900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:15.378082 6900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:15.384678 6900 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:44177
I20250114 03:05:15.384687 6966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:44177 every 8 connection(s)
I20250114 03:05:15.385684 6900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:15.388244 6967 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:15.390607 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6900
I20250114 03:05:15.390724 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:15.393981 6967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962: Bootstrap starting.
I20250114 03:05:15.396479 6967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:15.397415 6967 log.cc:826] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:15.399374 6967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962: No bootstrap required, opened a new log
I20250114 03:05:15.402500 6967 raft_consensus.cc:357] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ae850ae8e4607b86100b539404962" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 44177 } }
I20250114 03:05:15.402701 6967 raft_consensus.cc:383] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:15.402762 6967 raft_consensus.cc:738] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 332ae850ae8e4607b86100b539404962, State: Initialized, Role: FOLLOWER
I20250114 03:05:15.403195 6967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ae850ae8e4607b86100b539404962" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 44177 } }
I20250114 03:05:15.403314 6967 raft_consensus.cc:397] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:15.403364 6967 raft_consensus.cc:491] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:15.403482 6967 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:15.404301 6967 raft_consensus.cc:513] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ae850ae8e4607b86100b539404962" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 44177 } }
I20250114 03:05:15.404636 6967 leader_election.cc:304] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 332ae850ae8e4607b86100b539404962; no voters:
I20250114 03:05:15.404948 6967 leader_election.cc:290] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:15.405045 6972 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:15.405210 6972 raft_consensus.cc:695] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [term 1 LEADER]: Becoming Leader. State: Replica: 332ae850ae8e4607b86100b539404962, State: Running, Role: LEADER
I20250114 03:05:15.405540 6972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ae850ae8e4607b86100b539404962" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 44177 } }
I20250114 03:05:15.406160 6967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:15.407267 6973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "332ae850ae8e4607b86100b539404962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ae850ae8e4607b86100b539404962" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 44177 } } }
I20250114 03:05:15.407402 6973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:15.407773 6974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 332ae850ae8e4607b86100b539404962. Latest consensus state: current_term: 1 leader_uuid: "332ae850ae8e4607b86100b539404962" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "332ae850ae8e4607b86100b539404962" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 44177 } } }
I20250114 03:05:15.407857 6974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962 [sys.catalog]: This master's current role is: LEADER
W20250114 03:05:15.410691 6988 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 03:05:15.410768 6988 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 03:05:15.410845 6981 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:15.411576 6981 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:15.416723 6981 catalog_manager.cc:1348] Generated new cluster ID: a7b0af6325434870a515b85e2bb8499d
I20250114 03:05:15.416882 6981 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:15.432747 6981 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:15.434013 6981 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:15.444176 6981 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 332ae850ae8e4607b86100b539404962: Generated new TSK 0
I20250114 03:05:15.445087 6981 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:15.456833 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:44177
--builtin_ntp_servers=127.1.109.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250114 03:05:15.567584 6991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:15.567860 6991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:15.567914 6991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:15.567993 6991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:15.571380 6991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:15.571532 6991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:15.575632 6991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:44177
--never_fsync=true
--heap_profile_path=/tmp/kudu.6991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:15.576766 6991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:15.577946 6991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:15.586807 6997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.587419 6996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.588459 6999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:15.589350 6991 server_base.cc:1034] running on GCE node
I20250114 03:05:15.589783 6991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:15.590381 6991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:15.591549 6991 hybrid_clock.cc:648] HybridClock initialized: now 1736823915591531 us; error 31 us; skew 500 ppm
I20250114 03:05:15.593604 6991 webserver.cc:458] Webserver started at http://127.1.109.65:44613/ using document root <none> and password file <none>
I20250114 03:05:15.594182 6991 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:15.594242 6991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:15.594447 6991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:15.596151 6991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d5d49a4678334e788991b23be3d51c7c"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.596612 6991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d5d49a4678334e788991b23be3d51c7c"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.600080 6991 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250114 03:05:15.602285 7006 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.603416 6991 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:15.603525 6991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d5d49a4678334e788991b23be3d51c7c"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.603617 6991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:15.627516 6991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:15.628168 6991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:15.628366 6991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:15.628957 6991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:15.629909 6991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:15.629957 6991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.630016 6991 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:15.630043 6991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.638432 6991 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:42333
I20250114 03:05:15.638458 7119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:42333 every 8 connection(s)
I20250114 03:05:15.639437 6991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:15.643309 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 6991
I20250114 03:05:15.643421 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:15.645932 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:44177
--builtin_ntp_servers=127.1.109.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:15.649544 7120 heartbeater.cc:346] Connected to a master server at 127.1.109.126:44177
I20250114 03:05:15.649809 7120 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:15.650352 7120 heartbeater.cc:510] Master 127.1.109.126:44177 requested a full tablet report, sending...
I20250114 03:05:15.651885 6931 ts_manager.cc:194] Registered new tserver with Master: d5d49a4678334e788991b23be3d51c7c (127.1.109.65:42333)
I20250114 03:05:15.653117 6931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:43087
W20250114 03:05:15.756552 7123 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:15.756842 7123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:15.756884 7123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:15.756971 7123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:15.760077 7123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:15.760219 7123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:15.763916 7123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:44177
--never_fsync=true
--heap_profile_path=/tmp/kudu.7123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:15.765074 7123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:15.766227 7123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:15.774995 7129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.775887 7131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.776537 7128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:15.777505 7123 server_base.cc:1034] running on GCE node
I20250114 03:05:15.777923 7123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:15.778504 7123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:15.779665 7123 hybrid_clock.cc:648] HybridClock initialized: now 1736823915779642 us; error 37 us; skew 500 ppm
I20250114 03:05:15.781796 7123 webserver.cc:458] Webserver started at http://127.1.109.66:44933/ using document root <none> and password file <none>
I20250114 03:05:15.782398 7123 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:15.782454 7123 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:15.782635 7123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:15.784332 7123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "5081b4380e4e4d83bd3412521eb1272b"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.784798 7123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "5081b4380e4e4d83bd3412521eb1272b"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.788117 7123 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250114 03:05:15.790342 7137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.791368 7123 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:15.791473 7123 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "5081b4380e4e4d83bd3412521eb1272b"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.791564 7123 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:15.808153 7123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:15.808789 7123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:15.808985 7123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:15.809515 7123 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:15.810457 7123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:15.810501 7123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.810544 7123 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:15.810575 7123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.819167 7123 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:40563
I20250114 03:05:15.819192 7250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:40563 every 8 connection(s)
I20250114 03:05:15.820113 7123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:15.822469 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7123
I20250114 03:05:15.822608 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:15.824687 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:44177
--builtin_ntp_servers=127.1.109.84:45495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:15.829279 7251 heartbeater.cc:346] Connected to a master server at 127.1.109.126:44177
I20250114 03:05:15.829535 7251 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:15.830097 7251 heartbeater.cc:510] Master 127.1.109.126:44177 requested a full tablet report, sending...
I20250114 03:05:15.831130 6931 ts_manager.cc:194] Registered new tserver with Master: 5081b4380e4e4d83bd3412521eb1272b (127.1.109.66:40563)
I20250114 03:05:15.831774 6931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:42853
W20250114 03:05:15.932868 7254 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:15.933111 7254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:15.933137 7254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:15.933214 7254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:15.936280 7254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:15.936384 7254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:15.940047 7254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:44177
--never_fsync=true
--heap_profile_path=/tmp/kudu.7254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:15.941130 7254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:15.942289 7254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:15.951156 7259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.951854 7260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:15.952853 7262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:15.953626 7254 server_base.cc:1034] running on GCE node
I20250114 03:05:15.954017 7254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:15.954679 7254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:15.955834 7254 hybrid_clock.cc:648] HybridClock initialized: now 1736823915955813 us; error 35 us; skew 500 ppm
I20250114 03:05:15.957832 7254 webserver.cc:458] Webserver started at http://127.1.109.67:45613/ using document root <none> and password file <none>
I20250114 03:05:15.958449 7254 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:15.958505 7254 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:15.958686 7254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:15.960371 7254 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "92fe8988f28840d794550e27966be299"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.960887 7254 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "92fe8988f28840d794550e27966be299"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.964324 7254 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:15.966516 7268 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.967499 7254 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:15.967609 7254 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "92fe8988f28840d794550e27966be299"
format_stamp: "Formatted at 2025-01-14 03:05:15 on dist-test-slave-r16g"
I20250114 03:05:15.967706 7254 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:15.986426 7254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:15.987048 7254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:15.987200 7254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:15.987727 7254 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:15.988643 7254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:15.988690 7254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.988730 7254 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:15.988745 7254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:15.997174 7254 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:33665
I20250114 03:05:15.997195 7381 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:33665 every 8 connection(s)
I20250114 03:05:15.998124 7254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:16.000308 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7254
I20250114 03:05:16.000425 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.LeaderStepDown.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:16.006968 7382 heartbeater.cc:346] Connected to a master server at 127.1.109.126:44177
I20250114 03:05:16.007197 7382 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:16.007704 7382 heartbeater.cc:510] Master 127.1.109.126:44177 requested a full tablet report, sending...
I20250114 03:05:16.008719 6931 ts_manager.cc:194] Registered new tserver with Master: 92fe8988f28840d794550e27966be299 (127.1.109.67:33665)
I20250114 03:05:16.009276 6931 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:45581
I20250114 03:05:16.014308 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:16.022678 6931 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50280:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250114 03:05:16.025717 6931 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:05:16.042025 7054 tablet_service.cc:1467] Processing CreateTablet for tablet bc0874b3fa9c490caf0a8ed3be7b08e8 (DEFAULT_TABLE table=TestTable [id=fcdd63454c66451e9ca76f9e73698c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:16.042097 7185 tablet_service.cc:1467] Processing CreateTablet for tablet bc0874b3fa9c490caf0a8ed3be7b08e8 (DEFAULT_TABLE table=TestTable [id=fcdd63454c66451e9ca76f9e73698c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:16.042097 7316 tablet_service.cc:1467] Processing CreateTablet for tablet bc0874b3fa9c490caf0a8ed3be7b08e8 (DEFAULT_TABLE table=TestTable [id=fcdd63454c66451e9ca76f9e73698c51]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:16.042963 7054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0874b3fa9c490caf0a8ed3be7b08e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.042971 7185 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0874b3fa9c490caf0a8ed3be7b08e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.042971 7316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0874b3fa9c490caf0a8ed3be7b08e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.048797 7400 tablet_bootstrap.cc:492] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299: Bootstrap starting.
I20250114 03:05:16.048882 7401 tablet_bootstrap.cc:492] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: Bootstrap starting.
I20250114 03:05:16.049437 7399 tablet_bootstrap.cc:492] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c: Bootstrap starting.
I20250114 03:05:16.051214 7401 tablet_bootstrap.cc:654] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.051306 7399 tablet_bootstrap.cc:654] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.051708 7400 tablet_bootstrap.cc:654] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.052011 7399 log.cc:826] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:16.052031 7401 log.cc:826] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:16.052361 7400 log.cc:826] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:16.053853 7401 tablet_bootstrap.cc:492] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: No bootstrap required, opened a new log
I20250114 03:05:16.053849 7399 tablet_bootstrap.cc:492] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c: No bootstrap required, opened a new log
I20250114 03:05:16.053849 7400 tablet_bootstrap.cc:492] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299: No bootstrap required, opened a new log
I20250114 03:05:16.054034 7401 ts_tablet_manager.cc:1397] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:16.054034 7400 ts_tablet_manager.cc:1397] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250114 03:05:16.054034 7399 ts_tablet_manager.cc:1397] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:05:16.057265 7401 raft_consensus.cc:357] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.057466 7401 raft_consensus.cc:738] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5081b4380e4e4d83bd3412521eb1272b, State: Initialized, Role: FOLLOWER
I20250114 03:05:16.057791 7401 consensus_queue.cc:260] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.057860 7399 raft_consensus.cc:357] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.057860 7400 raft_consensus.cc:357] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.058063 7400 raft_consensus.cc:738] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92fe8988f28840d794550e27966be299, State: Initialized, Role: FOLLOWER
I20250114 03:05:16.058063 7399 raft_consensus.cc:738] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5d49a4678334e788991b23be3d51c7c, State: Initialized, Role: FOLLOWER
I20250114 03:05:16.058359 7399 consensus_queue.cc:260] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.058359 7400 consensus_queue.cc:260] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.058629 7401 ts_tablet_manager.cc:1428] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:16.059063 7382 heartbeater.cc:502] Master 127.1.109.126:44177 was elected leader, sending a full tablet report...
I20250114 03:05:16.059208 7399 ts_tablet_manager.cc:1428] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20250114 03:05:16.059280 7120 heartbeater.cc:502] Master 127.1.109.126:44177 was elected leader, sending a full tablet report...
I20250114 03:05:16.059983 7251 heartbeater.cc:502] Master 127.1.109.126:44177 was elected leader, sending a full tablet report...
I20250114 03:05:16.060057 7400 ts_tablet_manager.cc:1428] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
W20250114 03:05:16.071007 7252 tablet.cc:2367] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:16.074622 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:16.076053 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d5d49a4678334e788991b23be3d51c7c to finish bootstrapping
I20250114 03:05:16.081718 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 5081b4380e4e4d83bd3412521eb1272b to finish bootstrapping
I20250114 03:05:16.086290 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 92fe8988f28840d794550e27966be299 to finish bootstrapping
I20250114 03:05:16.095911 7074 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bc0874b3fa9c490caf0a8ed3be7b08e8"
dest_uuid: "d5d49a4678334e788991b23be3d51c7c"
from {username='slave'} at 127.0.0.1:59412
I20250114 03:05:16.096139 7074 raft_consensus.cc:491] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:16.096235 7074 raft_consensus.cc:3054] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:16.097270 7074 raft_consensus.cc:513] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.098280 7074 leader_election.cc:290] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [CANDIDATE]: Term 1 election: Requested vote from peers 5081b4380e4e4d83bd3412521eb1272b (127.1.109.66:40563), 92fe8988f28840d794550e27966be299 (127.1.109.67:33665)
I20250114 03:05:16.102769 7205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0874b3fa9c490caf0a8ed3be7b08e8" candidate_uuid: "d5d49a4678334e788991b23be3d51c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5081b4380e4e4d83bd3412521eb1272b"
I20250114 03:05:16.102900 7336 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc0874b3fa9c490caf0a8ed3be7b08e8" candidate_uuid: "d5d49a4678334e788991b23be3d51c7c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "92fe8988f28840d794550e27966be299"
I20250114 03:05:16.102981 7205 raft_consensus.cc:3054] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:16.103108 7336 raft_consensus.cc:3054] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:16.104178 7205 raft_consensus.cc:2463] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d49a4678334e788991b23be3d51c7c in term 1.
I20250114 03:05:16.104334 7336 raft_consensus.cc:2463] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5d49a4678334e788991b23be3d51c7c in term 1.
I20250114 03:05:16.104679 7008 leader_election.cc:304] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5081b4380e4e4d83bd3412521eb1272b, d5d49a4678334e788991b23be3d51c7c; no voters:
I20250114 03:05:16.105037 7407 raft_consensus.cc:2798] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:16.105244 7407 raft_consensus.cc:695] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 LEADER]: Becoming Leader. State: Replica: d5d49a4678334e788991b23be3d51c7c, State: Running, Role: LEADER
I20250114 03:05:16.105576 7407 consensus_queue.cc:237] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.108793 6929 catalog_manager.cc:5526] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c reported cstate change: term changed from 0 to 1, leader changed from <none> to d5d49a4678334e788991b23be3d51c7c (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d5d49a4678334e788991b23be3d51c7c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } health_report { overall_health: UNKNOWN } } }
I20250114 03:05:16.114459 7336 raft_consensus.cc:1270] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 92fe8988f28840d794550e27966be299 [term 1 FOLLOWER]: Refusing update from remote peer d5d49a4678334e788991b23be3d51c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:16.114459 7205 raft_consensus.cc:1270] T bc0874b3fa9c490caf0a8ed3be7b08e8 P 5081b4380e4e4d83bd3412521eb1272b [term 1 FOLLOWER]: Refusing update from remote peer d5d49a4678334e788991b23be3d51c7c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:16.115121 7411 consensus_queue.cc:1035] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:16.115293 7407 consensus_queue.cc:1035] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [LEADER]: Connected to new peer: Peer: permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:16.134155 7074 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc0874b3fa9c490caf0a8ed3be7b08e8"
dest_uuid: "d5d49a4678334e788991b23be3d51c7c"
from {username='slave'} at 127.0.0.1:59412
I20250114 03:05:16.134341 7074 raft_consensus.cc:589] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 LEADER]: Received request to step down
I20250114 03:05:16.134405 7074 raft_consensus.cc:3049] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:16.134465 7074 raft_consensus.cc:738] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d5d49a4678334e788991b23be3d51c7c, State: Running, Role: LEADER
I20250114 03:05:16.134629 7074 consensus_queue.cc:260] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5d49a4678334e788991b23be3d51c7c" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 42333 } } peers { permanent_uuid: "5081b4380e4e4d83bd3412521eb1272b" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 40563 } } peers { permanent_uuid: "92fe8988f28840d794550e27966be299" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 33665 } }
I20250114 03:05:16.134902 7074 raft_consensus.cc:3054] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:16.136162 7074 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bc0874b3fa9c490caf0a8ed3be7b08e8"
dest_uuid: "d5d49a4678334e788991b23be3d51c7c"
from {username='slave'} at 127.0.0.1:59412
I20250114 03:05:16.136267 7074 raft_consensus.cc:582] T bc0874b3fa9c490caf0a8ed3be7b08e8 P d5d49a4678334e788991b23be3d51c7c [term 2 FOLLOWER]: Rejecting request to step down while not leader
W20250114 03:05:16.137573 7034 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:59412: Illegal state: replica d5d49a4678334e788991b23be3d51c7c is not leader of this config: current role FOLLOWER
I20250114 03:05:16.138366 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6991
I20250114 03:05:16.145476 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7123
I20250114 03:05:16.151839 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7254
I20250114 03:05:16.158495 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 6900
2025-01-14T03:05:16Z chronyd exiting
[ OK ] RaftConsensusElectionITest.LeaderStepDown (964 ms)
[ RUN ] RaftConsensusElectionITest.StepDownWithSlowFollower
I20250114 03:05:16.180645 1461 test_util.cc:274] Using random seed: -728779616
I20250114 03:05:16.181486 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:16.181540 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:16.181551 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:05:16.181562 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:05:16.181572 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:16Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:16Z Disabled control of system clock
I20250114 03:05:16.189463 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:33763
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:33763
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250114 03:05:16.294044 7429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:16.294260 7429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:16.294299 7429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:16.297363 7429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:16.297417 7429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:16.297436 7429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:16.297456 7429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:16.301023 7429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:33763
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:33763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:16.301990 7429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:16.303046 7429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:16.310189 7437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.310837 7434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.311833 7435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:16.312664 7429 server_base.cc:1034] running on GCE node
I20250114 03:05:16.313195 7429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:16.314074 7429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:16.315237 7429 hybrid_clock.cc:648] HybridClock initialized: now 1736823916315218 us; error 40 us; skew 500 ppm
I20250114 03:05:16.317214 7429 webserver.cc:458] Webserver started at http://127.1.109.126:43533/ using document root <none> and password file <none>
I20250114 03:05:16.317780 7429 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:16.317833 7429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:16.318017 7429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:16.319706 7429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "c39283a8f76a48cebc4190f46e65c526"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.320183 7429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "c39283a8f76a48cebc4190f46e65c526"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.323657 7429 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:05:16.325901 7443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.326963 7429 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:16.327087 7429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "c39283a8f76a48cebc4190f46e65c526"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.327179 7429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:16.340503 7429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:16.341241 7429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:16.341408 7429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:16.348151 7495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:33763 every 8 connection(s)
I20250114 03:05:16.348152 7429 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:33763
I20250114 03:05:16.349254 7429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:16.351888 7496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.354193 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7429
I20250114 03:05:16.354313 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:16.357574 7496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526: Bootstrap starting.
I20250114 03:05:16.359673 7496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.360459 7496 log.cc:826] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:16.362249 7496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526: No bootstrap required, opened a new log
I20250114 03:05:16.365341 7496 raft_consensus.cc:357] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39283a8f76a48cebc4190f46e65c526" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 33763 } }
I20250114 03:05:16.365527 7496 raft_consensus.cc:383] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:16.365562 7496 raft_consensus.cc:738] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c39283a8f76a48cebc4190f46e65c526, State: Initialized, Role: FOLLOWER
I20250114 03:05:16.365937 7496 consensus_queue.cc:260] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39283a8f76a48cebc4190f46e65c526" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 33763 } }
I20250114 03:05:16.366042 7496 raft_consensus.cc:397] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:16.366086 7496 raft_consensus.cc:491] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:16.366163 7496 raft_consensus.cc:3054] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:16.366890 7496 raft_consensus.cc:513] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39283a8f76a48cebc4190f46e65c526" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 33763 } }
I20250114 03:05:16.367184 7496 leader_election.cc:304] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c39283a8f76a48cebc4190f46e65c526; no voters:
I20250114 03:05:16.367413 7496 leader_election.cc:290] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:16.367554 7501 raft_consensus.cc:2798] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:16.367765 7501 raft_consensus.cc:695] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [term 1 LEADER]: Becoming Leader. State: Replica: c39283a8f76a48cebc4190f46e65c526, State: Running, Role: LEADER
I20250114 03:05:16.368129 7501 consensus_queue.cc:237] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39283a8f76a48cebc4190f46e65c526" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 33763 } }
I20250114 03:05:16.368583 7496 sys_catalog.cc:564] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:16.369911 7502 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c39283a8f76a48cebc4190f46e65c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39283a8f76a48cebc4190f46e65c526" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 33763 } } }
I20250114 03:05:16.369906 7503 sys_catalog.cc:455] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c39283a8f76a48cebc4190f46e65c526. Latest consensus state: current_term: 1 leader_uuid: "c39283a8f76a48cebc4190f46e65c526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c39283a8f76a48cebc4190f46e65c526" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 33763 } } }
I20250114 03:05:16.370057 7502 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:16.370069 7503 sys_catalog.cc:458] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:16.370966 7511 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:16.373546 7511 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:16.379086 7511 catalog_manager.cc:1348] Generated new cluster ID: b1d5419ea1bf42e0813b9ec67245160d
I20250114 03:05:16.379158 7511 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:16.386015 7511 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:16.386945 7511 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:16.405615 7511 catalog_manager.cc:5899] T 00000000000000000000000000000000 P c39283a8f76a48cebc4190f46e65c526: Generated new TSK 0
I20250114 03:05:16.406388 7511 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:16.412575 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:33763
--builtin_ntp_servers=127.1.109.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
W20250114 03:05:16.523684 7520 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:16.523910 7520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:16.523932 7520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:16.524012 7520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:16.527208 7520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:16.527365 7520 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:16.531042 7520 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:33763
--never_fsync=true
--heap_profile_path=/tmp/kudu.7520
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:16.532176 7520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:16.533316 7520 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:16.541783 7526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.542343 7525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.543292 7528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:16.544121 7520 server_base.cc:1034] running on GCE node
I20250114 03:05:16.544512 7520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:16.545135 7520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:16.546301 7520 hybrid_clock.cc:648] HybridClock initialized: now 1736823916546265 us; error 36 us; skew 500 ppm
I20250114 03:05:16.548377 7520 webserver.cc:458] Webserver started at http://127.1.109.65:42381/ using document root <none> and password file <none>
I20250114 03:05:16.549034 7520 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:16.549095 7520 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:16.549273 7520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:16.551000 7520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "71388c0f85b14c87bd9123f4fd85f467"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.551460 7520 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "71388c0f85b14c87bd9123f4fd85f467"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.554814 7520 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:16.557018 7534 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.558045 7520 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:16.558158 7520 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "71388c0f85b14c87bd9123f4fd85f467"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.558241 7520 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:16.573129 7520 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:16.573769 7520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:16.573935 7520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:16.574527 7520 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:16.575448 7520 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:16.575500 7520 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.575562 7520 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:16.575587 7520 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.586323 7520 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:40397
I20250114 03:05:16.586351 7647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:40397 every 8 connection(s)
I20250114 03:05:16.587301 7520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:16.588443 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7520
I20250114 03:05:16.588559 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:16.590790 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:33763
--builtin_ntp_servers=127.1.109.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20250114 03:05:16.596872 7648 heartbeater.cc:346] Connected to a master server at 127.1.109.126:33763
I20250114 03:05:16.597141 7648 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:16.597851 7648 heartbeater.cc:510] Master 127.1.109.126:33763 requested a full tablet report, sending...
I20250114 03:05:16.599486 7460 ts_manager.cc:194] Registered new tserver with Master: 71388c0f85b14c87bd9123f4fd85f467 (127.1.109.65:40397)
I20250114 03:05:16.600761 7460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:54997
W20250114 03:05:16.701145 7651 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:16.701402 7651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:16.701424 7651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:16.701488 7651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:16.704672 7651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:16.704802 7651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:16.708628 7651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:33763
--never_fsync=true
--heap_profile_path=/tmp/kudu.7651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:16.709726 7651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:16.710886 7651 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:16.719173 7656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.719962 7659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.720757 7657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:16.721870 7651 server_base.cc:1034] running on GCE node
I20250114 03:05:16.722225 7651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:16.722772 7651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:16.723928 7651 hybrid_clock.cc:648] HybridClock initialized: now 1736823916723903 us; error 39 us; skew 500 ppm
I20250114 03:05:16.725751 7651 webserver.cc:458] Webserver started at http://127.1.109.66:39863/ using document root <none> and password file <none>
I20250114 03:05:16.726307 7651 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:16.726356 7651 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:16.726506 7651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:16.728117 7651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.728574 7651 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.731890 7651 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:16.733994 7665 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.735108 7651 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:16.735199 7651 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.735275 7651 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:16.749197 7651 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:16.750044 7651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:16.750259 7651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:16.750964 7651 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:16.751883 7651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:16.751935 7651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.751993 7651 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:16.752025 7651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.760445 7778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:46581 every 8 connection(s)
I20250114 03:05:16.760428 7651 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:46581
I20250114 03:05:16.761477 7651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:16.766852 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7651
I20250114 03:05:16.766971 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:16.769286 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:33763
--builtin_ntp_servers=127.1.109.84:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--consensus_rpc_timeout_ms=15000 with env {}
I20250114 03:05:16.770299 7779 heartbeater.cc:346] Connected to a master server at 127.1.109.126:33763
I20250114 03:05:16.770534 7779 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:16.771080 7779 heartbeater.cc:510] Master 127.1.109.126:33763 requested a full tablet report, sending...
I20250114 03:05:16.772032 7460 ts_manager.cc:194] Registered new tserver with Master: 10dcbcbb0c8441dbaf3a2795ecde42b8 (127.1.109.66:46581)
I20250114 03:05:16.772562 7460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:33495
W20250114 03:05:16.874987 7782 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:16.875221 7782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:16.875264 7782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:16.875339 7782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:16.878396 7782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:16.878530 7782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:16.882123 7782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--consensus_rpc_timeout_ms=15000
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:33763
--never_fsync=true
--heap_profile_path=/tmp/kudu.7782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:16.883206 7782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:16.884368 7782 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:16.893234 7788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.893864 7790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:16.894666 7787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:16.895563 7782 server_base.cc:1034] running on GCE node
I20250114 03:05:16.895980 7782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:16.896570 7782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:16.897719 7782 hybrid_clock.cc:648] HybridClock initialized: now 1736823916897703 us; error 30 us; skew 500 ppm
I20250114 03:05:16.899632 7782 webserver.cc:458] Webserver started at http://127.1.109.67:37473/ using document root <none> and password file <none>
I20250114 03:05:16.900207 7782 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:16.900266 7782 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:16.900431 7782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:16.902122 7782 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2aa15d918a3b4ca28e10c3ba20048fcb"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.902606 7782 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2aa15d918a3b4ca28e10c3ba20048fcb"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.905895 7782 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:05:16.907882 7796 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.908943 7782 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:16.909041 7782 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2aa15d918a3b4ca28e10c3ba20048fcb"
format_stamp: "Formatted at 2025-01-14 03:05:16 on dist-test-slave-r16g"
I20250114 03:05:16.909132 7782 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:16.927636 7782 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:16.928261 7782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:16.928407 7782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:16.929065 7782 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:16.929958 7782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:16.930002 7782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.930044 7782 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:16.930074 7782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:16.938983 7782 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:39747
I20250114 03:05:16.939002 7909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:39747 every 8 connection(s)
I20250114 03:05:16.940042 7782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:16.944993 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7782
I20250114 03:05:16.945106 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.StepDownWithSlowFollower.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:16.948750 7910 heartbeater.cc:346] Connected to a master server at 127.1.109.126:33763
I20250114 03:05:16.949011 7910 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:16.949546 7910 heartbeater.cc:510] Master 127.1.109.126:33763 requested a full tablet report, sending...
I20250114 03:05:16.950491 7460 ts_manager.cc:194] Registered new tserver with Master: 2aa15d918a3b4ca28e10c3ba20048fcb (127.1.109.67:39747)
I20250114 03:05:16.951001 7460 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:54581
I20250114 03:05:16.959297 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:16.968207 7460 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59180:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250114 03:05:16.970865 7460 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:05:16.986214 7844 tablet_service.cc:1467] Processing CreateTablet for tablet d881044cd8284bc98e968dd585add801 (DEFAULT_TABLE table=TestTable [id=7cb186638fc446e6bb9749f17c1c1063]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:16.986235 7713 tablet_service.cc:1467] Processing CreateTablet for tablet d881044cd8284bc98e968dd585add801 (DEFAULT_TABLE table=TestTable [id=7cb186638fc446e6bb9749f17c1c1063]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:16.986235 7582 tablet_service.cc:1467] Processing CreateTablet for tablet d881044cd8284bc98e968dd585add801 (DEFAULT_TABLE table=TestTable [id=7cb186638fc446e6bb9749f17c1c1063]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:16.987102 7844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d881044cd8284bc98e968dd585add801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.987103 7713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d881044cd8284bc98e968dd585add801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.987103 7582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d881044cd8284bc98e968dd585add801. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:16.992069 7927 tablet_bootstrap.cc:492] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb: Bootstrap starting.
I20250114 03:05:16.993294 7928 tablet_bootstrap.cc:492] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: Bootstrap starting.
I20250114 03:05:16.993635 7929 tablet_bootstrap.cc:492] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467: Bootstrap starting.
I20250114 03:05:16.994552 7927 tablet_bootstrap.cc:654] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.995204 7927 log.cc:826] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:16.995265 7929 tablet_bootstrap.cc:654] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.995888 7929 log.cc:826] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:16.996763 7927 tablet_bootstrap.cc:492] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb: No bootstrap required, opened a new log
I20250114 03:05:16.996958 7927 ts_tablet_manager.cc:1397] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:16.997493 7929 tablet_bootstrap.cc:492] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467: No bootstrap required, opened a new log
I20250114 03:05:16.997653 7929 ts_tablet_manager.cc:1397] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20250114 03:05:16.998116 7928 tablet_bootstrap.cc:654] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:16.998894 7928 log.cc:826] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:17.000159 7927 raft_consensus.cc:357] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.000330 7927 raft_consensus.cc:738] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aa15d918a3b4ca28e10c3ba20048fcb, State: Initialized, Role: FOLLOWER
I20250114 03:05:17.000543 7928 tablet_bootstrap.cc:492] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: No bootstrap required, opened a new log
I20250114 03:05:17.000716 7928 ts_tablet_manager.cc:1397] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250114 03:05:17.000654 7927 consensus_queue.cc:260] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.001315 7910 heartbeater.cc:502] Master 127.1.109.126:33763 was elected leader, sending a full tablet report...
I20250114 03:05:17.001315 7929 raft_consensus.cc:357] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.001504 7927 ts_tablet_manager.cc:1428] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250114 03:05:17.001514 7929 raft_consensus.cc:738] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71388c0f85b14c87bd9123f4fd85f467, State: Initialized, Role: FOLLOWER
I20250114 03:05:17.001811 7929 consensus_queue.cc:260] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.002394 7648 heartbeater.cc:502] Master 127.1.109.126:33763 was elected leader, sending a full tablet report...
I20250114 03:05:17.002640 7929 ts_tablet_manager.cc:1428] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:17.004770 7928 raft_consensus.cc:357] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.005013 7928 raft_consensus.cc:738] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 10dcbcbb0c8441dbaf3a2795ecde42b8, State: Initialized, Role: FOLLOWER
I20250114 03:05:17.005363 7928 consensus_queue.cc:260] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.006071 7779 heartbeater.cc:502] Master 127.1.109.126:33763 was elected leader, sending a full tablet report...
I20250114 03:05:17.006451 7928 ts_tablet_manager.cc:1428] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20250114 03:05:17.012276 7780 tablet.cc:2367] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:17.014021 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:17.015292 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 71388c0f85b14c87bd9123f4fd85f467 to finish bootstrapping
I20250114 03:05:17.020242 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 10dcbcbb0c8441dbaf3a2795ecde42b8 to finish bootstrapping
I20250114 03:05:17.024890 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 2aa15d918a3b4ca28e10c3ba20048fcb to finish bootstrapping
I20250114 03:05:17.032941 7733 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d881044cd8284bc98e968dd585add801"
dest_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8"
from {username='slave'} at 127.0.0.1:47102
I20250114 03:05:17.033142 7733 raft_consensus.cc:491] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:17.033241 7733 raft_consensus.cc:3054] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:17.034140 7733 raft_consensus.cc:513] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.035166 7733 leader_election.cc:290] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [CANDIDATE]: Term 1 election: Requested vote from peers 71388c0f85b14c87bd9123f4fd85f467 (127.1.109.65:40397), 2aa15d918a3b4ca28e10c3ba20048fcb (127.1.109.67:39747)
I20250114 03:05:17.038898 7864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d881044cd8284bc98e968dd585add801" candidate_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb"
I20250114 03:05:17.039104 7864 raft_consensus.cc:3054] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:17.040117 7864 raft_consensus.cc:2463] T d881044cd8284bc98e968dd585add801 P 2aa15d918a3b4ca28e10c3ba20048fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10dcbcbb0c8441dbaf3a2795ecde42b8 in term 1.
I20250114 03:05:17.040480 7602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d881044cd8284bc98e968dd585add801" candidate_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "71388c0f85b14c87bd9123f4fd85f467"
I20250114 03:05:17.040510 7666 leader_election.cc:304] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 10dcbcbb0c8441dbaf3a2795ecde42b8, 2aa15d918a3b4ca28e10c3ba20048fcb; no voters:
I20250114 03:05:17.040669 7602 raft_consensus.cc:3054] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:17.040793 7935 raft_consensus.cc:2798] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:17.041020 7935 raft_consensus.cc:695] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 LEADER]: Becoming Leader. State: Replica: 10dcbcbb0c8441dbaf3a2795ecde42b8, State: Running, Role: LEADER
I20250114 03:05:17.041335 7935 consensus_queue.cc:237] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:17.041792 7602 raft_consensus.cc:2463] T d881044cd8284bc98e968dd585add801 P 71388c0f85b14c87bd9123f4fd85f467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 10dcbcbb0c8441dbaf3a2795ecde42b8 in term 1.
I20250114 03:05:17.044705 7460 catalog_manager.cc:5526] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 10dcbcbb0c8441dbaf3a2795ecde42b8 (127.1.109.66). New cstate: current_term: 1 leader_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } health_report { overall_health: UNKNOWN } } }
I20250114 03:05:18.065711 7733 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d881044cd8284bc98e968dd585add801"
dest_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8"
from {username='slave'} at 127.0.0.1:47102
I20250114 03:05:18.065949 7733 raft_consensus.cc:589] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 LEADER]: Received request to step down
I20250114 03:05:18.066010 7733 raft_consensus.cc:3049] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:18.066040 7733 raft_consensus.cc:738] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 10dcbcbb0c8441dbaf3a2795ecde42b8, State: Running, Role: LEADER
I20250114 03:05:18.066190 7733 consensus_queue.cc:260] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "10dcbcbb0c8441dbaf3a2795ecde42b8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 46581 } } peers { permanent_uuid: "71388c0f85b14c87bd9123f4fd85f467" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 40397 } } peers { permanent_uuid: "2aa15d918a3b4ca28e10c3ba20048fcb" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39747 } }
I20250114 03:05:18.066334 7733 raft_consensus.cc:3054] T d881044cd8284bc98e968dd585add801 P 10dcbcbb0c8441dbaf3a2795ecde42b8 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:18.067056 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7520
W20250114 03:05:18.073560 7669 connection.cc:537] client connection to 127.1.109.65:40397 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:05:18.073755 7669 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 03:05:18.073797 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7651
I20250114 03:05:18.079087 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7782
I20250114 03:05:18.084759 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7429
2025-01-14T03:05:18Z chronyd exiting
[ OK ] RaftConsensusElectionITest.StepDownWithSlowFollower (1925 ms)
[ RUN ] RaftConsensusElectionITest.ElectPendingVoter
I20250114 03:05:18.105949 1461 test_util.cc:274] Using random seed: -726854312
I20250114 03:05:18.106666 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:18.106703 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:18.106715 1461 ts_itest-base.cc:114] 5 tablet servers
I20250114 03:05:18.106725 1461 ts_itest-base.cc:115] 5 replicas per TS
I20250114 03:05:18.106737 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:18Z Disabled control of system clock
I20250114 03:05:18.114449 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34883
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:34883
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250114 03:05:18.221916 7948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:18.222150 7948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:18.222221 7948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:18.225368 7948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:18.225427 7948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:18.225461 7948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:18.225492 7948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:18.229277 7948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:34883
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:34883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:18.230329 7948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:18.231446 7948 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:18.238998 7954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.239678 7956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.240721 7953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:18.241603 7948 server_base.cc:1034] running on GCE node
I20250114 03:05:18.242144 7948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:18.243085 7948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:18.244254 7948 hybrid_clock.cc:648] HybridClock initialized: now 1736823918244238 us; error 29 us; skew 500 ppm
I20250114 03:05:18.246284 7948 webserver.cc:458] Webserver started at http://127.1.109.126:38517/ using document root <none> and password file <none>
I20250114 03:05:18.246917 7948 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:18.246976 7948 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:18.247161 7948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:18.248816 7948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "53bf9f223cb14e7fb8993fadb2181812"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.249326 7948 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "53bf9f223cb14e7fb8993fadb2181812"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.252727 7948 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:18.254971 7962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.255967 7948 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:18.256083 7948 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "53bf9f223cb14e7fb8993fadb2181812"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.256170 7948 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:18.278235 7948 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:18.278967 7948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:18.279131 7948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:18.285771 7948 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:34883
I20250114 03:05:18.285781 8014 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:34883 every 8 connection(s)
I20250114 03:05:18.286772 7948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:18.289517 8015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:18.289532 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 7948
I20250114 03:05:18.290580 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:18.295970 8015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812: Bootstrap starting.
I20250114 03:05:18.298761 8015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:18.299515 8015 log.cc:826] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:18.301301 8015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812: No bootstrap required, opened a new log
I20250114 03:05:18.304144 8015 raft_consensus.cc:357] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bf9f223cb14e7fb8993fadb2181812" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34883 } }
I20250114 03:05:18.304324 8015 raft_consensus.cc:383] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:18.304360 8015 raft_consensus.cc:738] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53bf9f223cb14e7fb8993fadb2181812, State: Initialized, Role: FOLLOWER
I20250114 03:05:18.304783 8015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bf9f223cb14e7fb8993fadb2181812" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34883 } }
I20250114 03:05:18.304937 8015 raft_consensus.cc:397] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:18.304986 8015 raft_consensus.cc:491] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:18.305064 8015 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:18.305819 8015 raft_consensus.cc:513] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bf9f223cb14e7fb8993fadb2181812" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34883 } }
I20250114 03:05:18.306142 8015 leader_election.cc:304] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53bf9f223cb14e7fb8993fadb2181812; no voters:
I20250114 03:05:18.306475 8015 leader_election.cc:290] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:18.306851 8020 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:18.307073 8020 raft_consensus.cc:695] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [term 1 LEADER]: Becoming Leader. State: Replica: 53bf9f223cb14e7fb8993fadb2181812, State: Running, Role: LEADER
I20250114 03:05:18.307344 8020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bf9f223cb14e7fb8993fadb2181812" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34883 } }
I20250114 03:05:18.307722 8015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:18.309187 8021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53bf9f223cb14e7fb8993fadb2181812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bf9f223cb14e7fb8993fadb2181812" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34883 } } }
I20250114 03:05:18.309233 8022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53bf9f223cb14e7fb8993fadb2181812. Latest consensus state: current_term: 1 leader_uuid: "53bf9f223cb14e7fb8993fadb2181812" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53bf9f223cb14e7fb8993fadb2181812" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 34883 } } }
I20250114 03:05:18.309294 8021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:18.309350 8022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:18.309752 8030 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:18.312665 8030 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:18.317867 8030 catalog_manager.cc:1348] Generated new cluster ID: 434a976c49af4fd3b4a3a55a23db1199
I20250114 03:05:18.317934 8030 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:18.331537 8030 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:18.332402 8030 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:18.341547 8030 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 53bf9f223cb14e7fb8993fadb2181812: Generated new TSK 0
I20250114 03:05:18.342366 8030 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:18.353055 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--builtin_ntp_servers=127.1.109.84:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250114 03:05:18.460844 8039 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:18.461094 8039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:18.461140 8039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:18.461213 8039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:18.464346 8039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:18.464490 8039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:18.468300 8039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--never_fsync=true
--heap_profile_path=/tmp/kudu.8039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:18.469465 8039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:18.470640 8039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:18.479609 8045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.480194 8044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.481249 8047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:18.481997 8039 server_base.cc:1034] running on GCE node
I20250114 03:05:18.482371 8039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:18.482933 8039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:18.484082 8039 hybrid_clock.cc:648] HybridClock initialized: now 1736823918484065 us; error 32 us; skew 500 ppm
I20250114 03:05:18.485936 8039 webserver.cc:458] Webserver started at http://127.1.109.65:38223/ using document root <none> and password file <none>
I20250114 03:05:18.486479 8039 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:18.486538 8039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:18.486709 8039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:18.488371 8039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "82a15e2b425f47fc98888a5b23eb540e"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.488881 8039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "82a15e2b425f47fc98888a5b23eb540e"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.492019 8039 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20250114 03:05:18.494094 8053 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.495060 8039 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250114 03:05:18.495174 8039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "82a15e2b425f47fc98888a5b23eb540e"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.495254 8039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:18.515079 8039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:18.515686 8039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:18.515856 8039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:18.516364 8039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:18.517279 8039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:18.517328 8039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.517388 8039 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:18.517421 8039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.525867 8039 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:46667
I20250114 03:05:18.525897 8166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:46667 every 8 connection(s)
I20250114 03:05:18.526788 8039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:18.528775 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8039
I20250114 03:05:18.528918 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:18.530995 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--builtin_ntp_servers=127.1.109.84:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:18.536388 8167 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34883
I20250114 03:05:18.536630 8167 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:18.537187 8167 heartbeater.cc:510] Master 127.1.109.126:34883 requested a full tablet report, sending...
I20250114 03:05:18.538656 7979 ts_manager.cc:194] Registered new tserver with Master: 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65:46667)
I20250114 03:05:18.539840 7979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:50725
W20250114 03:05:18.634320 8170 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:18.634583 8170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:18.634624 8170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:18.634699 8170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:18.637912 8170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:18.638036 8170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:18.642129 8170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--never_fsync=true
--heap_profile_path=/tmp/kudu.8170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:18.643201 8170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:18.644335 8170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:18.652809 8178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.653663 8176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.654531 8175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:18.655426 8170 server_base.cc:1034] running on GCE node
I20250114 03:05:18.655833 8170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:18.656381 8170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:18.657536 8170 hybrid_clock.cc:648] HybridClock initialized: now 1736823918657519 us; error 45 us; skew 500 ppm
I20250114 03:05:18.659411 8170 webserver.cc:458] Webserver started at http://127.1.109.66:34463/ using document root <none> and password file <none>
I20250114 03:05:18.659983 8170 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:18.660036 8170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:18.660216 8170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:18.661963 8170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0563117b10aa414ebcc37f2441a8480a"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.662443 8170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0563117b10aa414ebcc37f2441a8480a"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.665900 8170 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20250114 03:05:18.668103 8184 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.669318 8170 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:18.669414 8170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0563117b10aa414ebcc37f2441a8480a"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.669515 8170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:18.700680 8170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:18.701373 8170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:18.701537 8170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:18.702060 8170 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:18.702967 8170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:18.703017 8170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.703081 8170 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:18.703107 8170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.712560 8170 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:44143
I20250114 03:05:18.712599 8297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:44143 every 8 connection(s)
I20250114 03:05:18.713670 8170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:18.716624 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8170
I20250114 03:05:18.716756 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:18.719260 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--builtin_ntp_servers=127.1.109.84:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:18.723400 8298 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34883
I20250114 03:05:18.723620 8298 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:18.724154 8298 heartbeater.cc:510] Master 127.1.109.126:34883 requested a full tablet report, sending...
I20250114 03:05:18.725298 7979 ts_manager.cc:194] Registered new tserver with Master: 0563117b10aa414ebcc37f2441a8480a (127.1.109.66:44143)
I20250114 03:05:18.726138 7979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:53823
W20250114 03:05:18.827586 8301 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:18.827803 8301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:18.827824 8301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:18.827862 8301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:18.830881 8301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:18.830981 8301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:18.834625 8301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--never_fsync=true
--heap_profile_path=/tmp/kudu.8301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:18.835633 8301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:18.836771 8301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:18.845716 8306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.846688 8307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:18.847240 8309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:18.848129 8301 server_base.cc:1034] running on GCE node
I20250114 03:05:18.848598 8301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:18.849229 8301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:18.850404 8301 hybrid_clock.cc:648] HybridClock initialized: now 1736823918850380 us; error 43 us; skew 500 ppm
I20250114 03:05:18.852262 8301 webserver.cc:458] Webserver started at http://127.1.109.67:39913/ using document root <none> and password file <none>
I20250114 03:05:18.852900 8301 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:18.852967 8301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:18.853152 8301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:18.854878 8301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7770a0a9e66f49dbb4b3d010905955a5"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.855355 8301 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7770a0a9e66f49dbb4b3d010905955a5"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.858744 8301 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:18.861055 8315 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.862098 8301 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:18.862213 8301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7770a0a9e66f49dbb4b3d010905955a5"
format_stamp: "Formatted at 2025-01-14 03:05:18 on dist-test-slave-r16g"
I20250114 03:05:18.862296 8301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:18.878589 8301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:18.879135 8301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:18.879285 8301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:18.879840 8301 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:18.880715 8301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:18.880761 8301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.880811 8301 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:18.880860 8301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:18.889138 8301 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:39325
I20250114 03:05:18.889253 8428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:39325 every 8 connection(s)
I20250114 03:05:18.890064 8301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:18.895064 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8301
I20250114 03:05:18.895164 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:18.897542 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.68:0
--local_ip_for_outbound_sockets=127.1.109.68
--webserver_interface=127.1.109.68
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--builtin_ntp_servers=127.1.109.84:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:18.898605 8429 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34883
I20250114 03:05:18.898855 8429 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:18.899399 8429 heartbeater.cc:510] Master 127.1.109.126:34883 requested a full tablet report, sending...
I20250114 03:05:18.900373 7979 ts_manager.cc:194] Registered new tserver with Master: 7770a0a9e66f49dbb4b3d010905955a5 (127.1.109.67:39325)
I20250114 03:05:18.900943 7979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:55065
W20250114 03:05:19.002295 8432 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:19.002539 8432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:19.002562 8432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:19.002609 8432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:19.005755 8432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:19.005867 8432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.68
I20250114 03:05:19.010116 8432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.1.109.68
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--never_fsync=true
--heap_profile_path=/tmp/kudu.8432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.68
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:19.011231 8432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:19.012333 8432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:19.020659 8437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:19.021232 8438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:19.022305 8440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:19.023236 8432 server_base.cc:1034] running on GCE node
I20250114 03:05:19.023682 8432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:19.024261 8432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:19.025411 8432 hybrid_clock.cc:648] HybridClock initialized: now 1736823919025388 us; error 45 us; skew 500 ppm
I20250114 03:05:19.031723 8432 webserver.cc:458] Webserver started at http://127.1.109.68:34695/ using document root <none> and password file <none>
I20250114 03:05:19.032403 8432 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:19.032473 8432 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:19.032675 8432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:19.034682 8432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e14e86bb6de0414ca913b5b9be8a8dc3"
format_stamp: "Formatted at 2025-01-14 03:05:19 on dist-test-slave-r16g"
I20250114 03:05:19.035195 8432 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e14e86bb6de0414ca913b5b9be8a8dc3"
format_stamp: "Formatted at 2025-01-14 03:05:19 on dist-test-slave-r16g"
I20250114 03:05:19.038892 8432 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:19.041234 8446 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:19.042344 8432 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:19.042495 8432 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e14e86bb6de0414ca913b5b9be8a8dc3"
format_stamp: "Formatted at 2025-01-14 03:05:19 on dist-test-slave-r16g"
I20250114 03:05:19.042613 8432 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:19.059515 8432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:19.060120 8432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:19.060250 8432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:19.060791 8432 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:19.061731 8432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:19.061784 8432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:19.061823 8432 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:19.061843 8432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:19.070384 8432 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.68:42917
I20250114 03:05:19.070395 8559 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.68:42917 every 8 connection(s)
I20250114 03:05:19.071339 8432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250114 03:05:19.073390 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8432
I20250114 03:05:19.073501 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250114 03:05:19.075692 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.69:0
--local_ip_for_outbound_sockets=127.1.109.69
--webserver_interface=127.1.109.69
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--builtin_ntp_servers=127.1.109.84:45609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:19.080809 8560 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34883
I20250114 03:05:19.081059 8560 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:19.081555 8560 heartbeater.cc:510] Master 127.1.109.126:34883 requested a full tablet report, sending...
I20250114 03:05:19.082563 7979 ts_manager.cc:194] Registered new tserver with Master: e14e86bb6de0414ca913b5b9be8a8dc3 (127.1.109.68:42917)
I20250114 03:05:19.083130 7979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.68:47809
W20250114 03:05:19.184346 8563 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:19.184604 8563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:19.184650 8563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:19.184727 8563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:19.187696 8563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:19.187839 8563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.69
I20250114 03:05:19.191531 8563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:45609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.1.109.69
--webserver_port=0
--tserver_master_addrs=127.1.109.126:34883
--never_fsync=true
--heap_profile_path=/tmp/kudu.8563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.69
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:19.192662 8563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:19.193783 8563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:19.202281 8568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:19.202811 8569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:19.203816 8571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:19.204680 8563 server_base.cc:1034] running on GCE node
I20250114 03:05:19.205112 8563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:19.205686 8563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:19.206831 8563 hybrid_clock.cc:648] HybridClock initialized: now 1736823919206810 us; error 37 us; skew 500 ppm
I20250114 03:05:19.208745 8563 webserver.cc:458] Webserver started at http://127.1.109.69:39957/ using document root <none> and password file <none>
I20250114 03:05:19.209401 8563 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:19.209470 8563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:19.209656 8563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:19.211398 8563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "0fa50a1adc224a84ad007d90a4998e20"
format_stamp: "Formatted at 2025-01-14 03:05:19 on dist-test-slave-r16g"
I20250114 03:05:19.211911 8563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "0fa50a1adc224a84ad007d90a4998e20"
format_stamp: "Formatted at 2025-01-14 03:05:19 on dist-test-slave-r16g"
I20250114 03:05:19.215265 8563 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20250114 03:05:19.217398 8577 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:19.218421 8563 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:19.218536 8563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "0fa50a1adc224a84ad007d90a4998e20"
format_stamp: "Formatted at 2025-01-14 03:05:19 on dist-test-slave-r16g"
I20250114 03:05:19.218638 8563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:19.238929 8563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:19.239563 8563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:19.239717 8563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:19.240269 8563 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:19.241312 8563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:19.241360 8563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:19.241406 8563 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:19.241437 8563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:19.250023 8563 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.69:34351
I20250114 03:05:19.250044 8690 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.69:34351 every 8 connection(s)
I20250114 03:05:19.251029 8563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250114 03:05:19.252478 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8563
I20250114 03:05:19.252583 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.ElectPendingVoter.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250114 03:05:19.259829 8691 heartbeater.cc:346] Connected to a master server at 127.1.109.126:34883
I20250114 03:05:19.260069 8691 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:19.260577 8691 heartbeater.cc:510] Master 127.1.109.126:34883 requested a full tablet report, sending...
I20250114 03:05:19.261610 7979 ts_manager.cc:194] Registered new tserver with Master: 0fa50a1adc224a84ad007d90a4998e20 (127.1.109.69:34351)
I20250114 03:05:19.262147 7979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.69:40231
I20250114 03:05:19.266727 1461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250114 03:05:19.274948 7979 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44998:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250114 03:05:19.277483 7979 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 6 tablet servers would be needed, 5 are available. Consider bringing up more tablet servers.
I20250114 03:05:19.294809 8232 tablet_service.cc:1467] Processing CreateTablet for tablet 10d01b01e23747d2bb46a1edb98b27b5 (DEFAULT_TABLE table=TestTable [id=d79faa54613943309c3a7c8f0be14d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:19.294816 8363 tablet_service.cc:1467] Processing CreateTablet for tablet 10d01b01e23747d2bb46a1edb98b27b5 (DEFAULT_TABLE table=TestTable [id=d79faa54613943309c3a7c8f0be14d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:19.295804 8232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d01b01e23747d2bb46a1edb98b27b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:19.295797 8363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d01b01e23747d2bb46a1edb98b27b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:19.295663 8101 tablet_service.cc:1467] Processing CreateTablet for tablet 10d01b01e23747d2bb46a1edb98b27b5 (DEFAULT_TABLE table=TestTable [id=d79faa54613943309c3a7c8f0be14d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:19.296386 8101 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d01b01e23747d2bb46a1edb98b27b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:19.303085 8713 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: Bootstrap starting.
I20250114 03:05:19.304257 8714 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: Bootstrap starting.
I20250114 03:05:19.305428 8713 tablet_bootstrap.cc:654] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:19.306283 8713 log.cc:826] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:19.306805 8714 tablet_bootstrap.cc:654] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:19.307581 8714 log.cc:826] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:19.308202 8713 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: No bootstrap required, opened a new log
I20250114 03:05:19.308400 8713 ts_tablet_manager.cc:1397] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:19.308869 8712 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: Bootstrap starting.
I20250114 03:05:19.309448 8714 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: No bootstrap required, opened a new log
I20250114 03:05:19.309636 8714 ts_tablet_manager.cc:1397] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20250114 03:05:19.311203 8712 tablet_bootstrap.cc:654] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:19.311993 8712 log.cc:826] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:19.312557 8713 raft_consensus.cc:357] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.313625 8714 raft_consensus.cc:357] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.313781 8713 raft_consensus.cc:738] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7770a0a9e66f49dbb4b3d010905955a5, State: Initialized, Role: FOLLOWER
I20250114 03:05:19.313897 8714 raft_consensus.cc:738] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82a15e2b425f47fc98888a5b23eb540e, State: Initialized, Role: FOLLOWER
I20250114 03:05:19.313582 8625 tablet_service.cc:1467] Processing CreateTablet for tablet 10d01b01e23747d2bb46a1edb98b27b5 (DEFAULT_TABLE table=TestTable [id=d79faa54613943309c3a7c8f0be14d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:19.314236 8714 consensus_queue.cc:260] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.314280 8713 consensus_queue.cc:260] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.314632 8625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d01b01e23747d2bb46a1edb98b27b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:19.314683 8494 tablet_service.cc:1467] Processing CreateTablet for tablet 10d01b01e23747d2bb46a1edb98b27b5 (DEFAULT_TABLE table=TestTable [id=d79faa54613943309c3a7c8f0be14d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:19.315140 8714 ts_tablet_manager.cc:1428] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20250114 03:05:19.315451 8713 ts_tablet_manager.cc:1428] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 03:05:19.315562 8494 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10d01b01e23747d2bb46a1edb98b27b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:19.315625 8167 heartbeater.cc:502] Master 127.1.109.126:34883 was elected leader, sending a full tablet report...
I20250114 03:05:19.316679 8429 heartbeater.cc:502] Master 127.1.109.126:34883 was elected leader, sending a full tablet report...
I20250114 03:05:19.317415 8712 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: No bootstrap required, opened a new log
I20250114 03:05:19.317595 8712 ts_tablet_manager.cc:1397] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20250114 03:05:19.322067 8712 raft_consensus.cc:357] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.322484 8712 raft_consensus.cc:738] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0563117b10aa414ebcc37f2441a8480a, State: Initialized, Role: FOLLOWER
I20250114 03:05:19.322842 8712 consensus_queue.cc:260] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.323793 8712 ts_tablet_manager.cc:1428] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 03:05:19.324108 8298 heartbeater.cc:502] Master 127.1.109.126:34883 was elected leader, sending a full tablet report...
I20250114 03:05:19.324429 8720 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: Bootstrap starting.
I20250114 03:05:19.325192 8721 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: Bootstrap starting.
I20250114 03:05:19.326735 8720 tablet_bootstrap.cc:654] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:19.327311 8721 tablet_bootstrap.cc:654] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:19.327447 1461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250114 03:05:19.327589 8720 log.cc:826] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:19.328068 8721 log.cc:826] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:19.329032 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 82a15e2b425f47fc98888a5b23eb540e to finish bootstrapping
I20250114 03:05:19.329370 8720 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: No bootstrap required, opened a new log
I20250114 03:05:19.329552 8720 ts_tablet_manager.cc:1397] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:19.329794 8721 tablet_bootstrap.cc:492] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: No bootstrap required, opened a new log
I20250114 03:05:19.329968 8721 ts_tablet_manager.cc:1397] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250114 03:05:19.333302 8720 raft_consensus.cc:357] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.333302 8721 raft_consensus.cc:357] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.333537 8720 raft_consensus.cc:738] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa50a1adc224a84ad007d90a4998e20, State: Initialized, Role: FOLLOWER
I20250114 03:05:19.333544 8721 raft_consensus.cc:738] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e14e86bb6de0414ca913b5b9be8a8dc3, State: Initialized, Role: FOLLOWER
I20250114 03:05:19.333853 8720 consensus_queue.cc:260] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.333866 8721 consensus_queue.cc:260] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.334579 8560 heartbeater.cc:502] Master 127.1.109.126:34883 was elected leader, sending a full tablet report...
I20250114 03:05:19.334792 8721 ts_tablet_manager.cc:1428] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20250114 03:05:19.334836 8720 ts_tablet_manager.cc:1428] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:19.335152 8691 heartbeater.cc:502] Master 127.1.109.126:34883 was elected leader, sending a full tablet report...
I20250114 03:05:19.336158 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0563117b10aa414ebcc37f2441a8480a to finish bootstrapping
I20250114 03:05:19.341912 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7770a0a9e66f49dbb4b3d010905955a5 to finish bootstrapping
I20250114 03:05:19.346678 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e14e86bb6de0414ca913b5b9be8a8dc3 to finish bootstrapping
I20250114 03:05:19.351372 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0fa50a1adc224a84ad007d90a4998e20 to finish bootstrapping
I20250114 03:05:19.359899 8252 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5"
dest_uuid: "0563117b10aa414ebcc37f2441a8480a"
from {username='slave'} at 127.0.0.1:60240
I20250114 03:05:19.360148 8252 raft_consensus.cc:491] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:19.360248 8252 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:19.361230 8252 raft_consensus.cc:513] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.362380 8252 leader_election.cc:290] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [CANDIDATE]: Term 1 election: Requested vote from peers 7770a0a9e66f49dbb4b3d010905955a5 (127.1.109.67:39325), 0fa50a1adc224a84ad007d90a4998e20 (127.1.109.69:34351), e14e86bb6de0414ca913b5b9be8a8dc3 (127.1.109.68:42917), 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65:46667)
I20250114 03:05:19.367236 8514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "0563117b10aa414ebcc37f2441a8480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3"
I20250114 03:05:19.367475 8514 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:19.368127 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "0563117b10aa414ebcc37f2441a8480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0fa50a1adc224a84ad007d90a4998e20"
I20250114 03:05:19.368211 8383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "0563117b10aa414ebcc37f2441a8480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7770a0a9e66f49dbb4b3d010905955a5"
I20250114 03:05:19.368346 8645 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:19.368367 8383 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:19.368752 8514 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0563117b10aa414ebcc37f2441a8480a in term 1.
I20250114 03:05:19.369611 8645 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0563117b10aa414ebcc37f2441a8480a in term 1.
I20250114 03:05:19.369607 8383 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0563117b10aa414ebcc37f2441a8480a in term 1.
I20250114 03:05:19.370074 8187 leader_election.cc:304] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0563117b10aa414ebcc37f2441a8480a, 0fa50a1adc224a84ad007d90a4998e20, e14e86bb6de0414ca913b5b9be8a8dc3; no voters:
I20250114 03:05:19.370258 8121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "0563117b10aa414ebcc37f2441a8480a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "82a15e2b425f47fc98888a5b23eb540e"
I20250114 03:05:19.370467 8722 raft_consensus.cc:2798] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:19.370462 8121 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:19.370693 8722 raft_consensus.cc:695] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 LEADER]: Becoming Leader. State: Replica: 0563117b10aa414ebcc37f2441a8480a, State: Running, Role: LEADER
I20250114 03:05:19.371060 8722 consensus_queue.cc:237] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }
I20250114 03:05:19.371403 8121 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0563117b10aa414ebcc37f2441a8480a in term 1.
I20250114 03:05:19.374923 7977 catalog_manager.cc:5526] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a reported cstate change: term changed from 0 to 1, leader changed from <none> to 0563117b10aa414ebcc37f2441a8480a (127.1.109.66). New cstate: current_term: 1 leader_uuid: "0563117b10aa414ebcc37f2441a8480a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } health_report { overall_health: UNKNOWN } } }
W20250114 03:05:19.391152 8430 tablet.cc:2367] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:05:19.465029 8299 tablet.cc:2367] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:05:19.501893 8692 tablet.cc:2367] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:05:19.528183 8168 tablet.cc:2367] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:05:19.572556 8561 tablet.cc:2367] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:19.797629 8722 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:19.835748 8722 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:19.841964 8722 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:19.842303 8722 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:19.844596 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 7948
W20250114 03:05:19.852563 8319 connection.cc:537] client connection to 127.1.109.126:34883 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:05:19.852563 8450 connection.cc:537] client connection to 127.1.109.126:34883 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:05:19.852864 8429 heartbeater.cc:643] Failed to heartbeat to 127.1.109.126:34883 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250114 03:05:19.853029 8560 heartbeater.cc:643] Failed to heartbeat to 127.1.109.126:34883 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250114 03:05:19.853462 1461 raft_consensus_election-itest.cc:666] Removing tserver with uuid 82a15e2b425f47fc98888a5b23eb540e
I20250114 03:05:19.854739 8252 consensus_queue.cc:237] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } }
I20250114 03:05:19.856148 8514 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:19.856218 8383 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:19.856220 8645 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:19.856544 8739 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:05:19.856693 8740 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:05:19.856866 8722 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:05:19.858676 8722 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } }
I20250114 03:05:19.859215 8645 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } }
I20250114 03:05:19.859505 8514 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } }
W20250114 03:05:19.862193 8298 heartbeater.cc:643] Failed to heartbeat to 127.1.109.126:34883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.109.126:34883: connect: Connection refused (error 111)
I20250114 03:05:19.862376 8383 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } }
W20250114 03:05:19.862847 8691 heartbeater.cc:643] Failed to heartbeat to 127.1.109.126:34883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.109.126:34883: connect: Connection refused (error 111)
I20250114 03:05:19.875394 1461 raft_consensus_election-itest.cc:676] Pausing 3 replicas...
I20250114 03:05:19.907238 1461 raft_consensus_election-itest.cc:684] Adding back Peer 82a15e2b425f47fc98888a5b23eb540e and expecting timeout...
I20250114 03:05:19.908509 8252 consensus_queue.cc:237] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } }
I20250114 03:05:19.910130 8121 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 03:05:19.910578 8733 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:05:19.912091 8121 raft_consensus.cc:867] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } }
W20250114 03:05:19.912179 8121 raft_consensus.cc:1574] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 3' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } }.
Pending config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } }:
New config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } }
I20250114 03:05:19.912581 8121 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.2: config changed from index -1 to 2, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) evicted. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } }
I20250114 03:05:19.914942 8121 raft_consensus.cc:1476] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 NON_PARTICIPANT]: Allowing update even though not a member of the config
W20250114 03:05:19.915469 8167 heartbeater.cc:643] Failed to heartbeat to 127.1.109.126:34883 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.1.109.126:34883: connect: Connection refused (error 111)
I20250114 03:05:20.008248 1461 raft_consensus_election-itest.cc:688] Timeout achieved.
I20250114 03:05:20.013515 1461 raft_consensus_election-itest.cc:699] Forcing Peer 0563117b10aa414ebcc37f2441a8480a to step down...
I20250114 03:05:20.014129 8252 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5"
dest_uuid: "0563117b10aa414ebcc37f2441a8480a"
from {username='slave'} at 127.0.0.1:60240
I20250114 03:05:20.014308 8252 raft_consensus.cc:589] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 LEADER]: Received request to step down
I20250114 03:05:20.014350 8252 raft_consensus.cc:3049] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:20.014391 8252 raft_consensus.cc:738] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0563117b10aa414ebcc37f2441a8480a, State: Running, Role: LEADER
I20250114 03:05:20.014565 8252 consensus_queue.cc:260] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } }
I20250114 03:05:20.014724 8252 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:20.015074 1461 raft_consensus_election-itest.cc:705] Resuming Peer 7770a0a9e66f49dbb4b3d010905955a5 ...
I20250114 03:05:20.016060 8383 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 03:05:20.016651 1461 raft_consensus_election-itest.cc:710] Trying to elect Peer 82a15e2b425f47fc98888a5b23eb540e ...
I20250114 03:05:20.017160 8121 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5"
dest_uuid: "82a15e2b425f47fc98888a5b23eb540e"
from {username='slave'} at 127.0.0.1:38698
I20250114 03:05:20.017338 8121 raft_consensus.cc:491] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:20.017401 8121 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:20.018443 8121 raft_consensus.cc:513] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } }
I20250114 03:05:20.020453 8121 leader_election.cc:290] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [CANDIDATE]: Term 2 election: Requested vote from peers 0563117b10aa414ebcc37f2441a8480a (127.1.109.66:44143), 7770a0a9e66f49dbb4b3d010905955a5 (127.1.109.67:39325), 0fa50a1adc224a84ad007d90a4998e20 (127.1.109.69:34351), e14e86bb6de0414ca913b5b9be8a8dc3 (127.1.109.68:42917)
I20250114 03:05:20.022796 1461 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 1.3
I20250114 03:05:20.023996 8383 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "82a15e2b425f47fc98888a5b23eb540e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "7770a0a9e66f49dbb4b3d010905955a5"
I20250114 03:05:20.024103 8383 raft_consensus.cc:1808] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 1 FOLLOWER]: Handling vote request from an unknown peer 82a15e2b425f47fc98888a5b23eb540e
I20250114 03:05:20.024156 8383 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:20.025135 8383 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a15e2b425f47fc98888a5b23eb540e in term 2.
I20250114 03:05:20.025382 8252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "82a15e2b425f47fc98888a5b23eb540e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0563117b10aa414ebcc37f2441a8480a"
I20250114 03:05:20.026237 8252 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a15e2b425f47fc98888a5b23eb540e in term 2.
I20250114 03:05:20.026593 8054 leader_election.cc:304] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 0563117b10aa414ebcc37f2441a8480a, 7770a0a9e66f49dbb4b3d010905955a5, 82a15e2b425f47fc98888a5b23eb540e; no voters:
I20250114 03:05:20.026861 8718 raft_consensus.cc:2798] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:05:20.027081 8718 raft_consensus.cc:695] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 2 LEADER]: Becoming Leader. State: Replica: 82a15e2b425f47fc98888a5b23eb540e, State: Running, Role: LEADER
I20250114 03:05:20.027410 8718 consensus_queue.cc:237] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } }
I20250114 03:05:20.125978 1461 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20250114 03:05:20.328585 1461 cluster_itest_util.cc:256] Not converged past 4 yet: 1.2 1.3 2.4
I20250114 03:05:20.408391 8252 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 2 FOLLOWER]: Refusing update from remote peer 82a15e2b425f47fc98888a5b23eb540e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250114 03:05:20.409077 8718 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 03:05:20.479225 8383 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 2 FOLLOWER]: Refusing update from remote peer 82a15e2b425f47fc98888a5b23eb540e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250114 03:05:20.479785 8718 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 03:05:20.482017 8761 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [term 2 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } } }
I20250114 03:05:20.482453 8383 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 7770a0a9e66f49dbb4b3d010905955a5 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } } }
I20250114 03:05:20.484900 8252 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 0563117b10aa414ebcc37f2441a8480a [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } } }
W20250114 03:05:20.486035 8252 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:60240 (request call id 47) took 578 ms (client timeout 100 ms). Trace:
W20250114 03:05:20.486114 8252 rpcz_store.cc:269] 0114 03:05:19.907927 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:19.908012 (+ 85us) service_pool.cc:225] Handling call
0114 03:05:19.909263 (+ 1251us) log.cc:844] Serialized 600 byte log entry
0114 03:05:20.486007 (+576744us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":235,"raft.queue_time_us":357,"raft.run_cpu_time_us":478,"raft.run_wall_time_us":480,"spinlock_wait_cycles":85760}
I20250114 03:05:20.631639 1461 raft_consensus_election-itest.cc:716] Resuming remaining nodes...
I20250114 03:05:20.633504 8514 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 03:05:20.634114 8645 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 1 FOLLOWER]: Refusing update from remote peer 0563117b10aa414ebcc37f2441a8480a: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20250114 03:05:20.638661 8644 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:20.639585 8645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "82a15e2b425f47fc98888a5b23eb540e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "0fa50a1adc224a84ad007d90a4998e20"
I20250114 03:05:20.639925 8644 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 2 FOLLOWER]: Refusing update from remote peer 82a15e2b425f47fc98888a5b23eb540e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250114 03:05:20.640106 8645 raft_consensus.cc:1808] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 2 FOLLOWER]: Handling vote request from an unknown peer 82a15e2b425f47fc98888a5b23eb540e
I20250114 03:05:20.641072 8645 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a15e2b425f47fc98888a5b23eb540e in term 2.
I20250114 03:05:20.641757 8768 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 03:05:20.644502 8645 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P 0fa50a1adc224a84ad007d90a4998e20 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } } }
I20250114 03:05:20.644953 8513 raft_consensus.cc:3054] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:20.646015 8513 raft_consensus.cc:1270] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 2 FOLLOWER]: Refusing update from remote peer 82a15e2b425f47fc98888a5b23eb540e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20250114 03:05:20.646414 8514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10d01b01e23747d2bb46a1edb98b27b5" candidate_uuid: "82a15e2b425f47fc98888a5b23eb540e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3"
I20250114 03:05:20.646428 8768 consensus_queue.cc:1035] T 10d01b01e23747d2bb46a1edb98b27b5 P 82a15e2b425f47fc98888a5b23eb540e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250114 03:05:20.646524 8514 raft_consensus.cc:1808] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 2 FOLLOWER]: Handling vote request from an unknown peer 82a15e2b425f47fc98888a5b23eb540e
I20250114 03:05:20.647432 8514 raft_consensus.cc:2463] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82a15e2b425f47fc98888a5b23eb540e in term 2.
I20250114 03:05:20.651331 8773 mvcc.cc:204] Tried to move back new op lower bound from 7114030778932461568 to 7114030776436178944. Current Snapshot: MvccSnapshot[applied={T|T < 7114030778932461568}]
I20250114 03:05:20.651634 8513 raft_consensus.cc:2949] T 10d01b01e23747d2bb46a1edb98b27b5 P e14e86bb6de0414ca913b5b9be8a8dc3 [term 2 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, VOTER 82a15e2b425f47fc98888a5b23eb540e (127.1.109.65) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0563117b10aa414ebcc37f2441a8480a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 44143 } } peers { permanent_uuid: "7770a0a9e66f49dbb4b3d010905955a5" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 39325 } } peers { permanent_uuid: "0fa50a1adc224a84ad007d90a4998e20" member_type: VOTER last_known_addr { host: "127.1.109.69" port: 34351 } } peers { permanent_uuid: "e14e86bb6de0414ca913b5b9be8a8dc3" member_type: VOTER last_known_addr { host: "127.1.109.68" port: 42917 } } peers { permanent_uuid: "82a15e2b425f47fc98888a5b23eb540e" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 46667 } attrs { } } }
I20250114 03:05:20.654641 8781 mvcc.cc:204] Tried to move back new op lower bound from 7114030778932461568 to 7114030776436178944. Current Snapshot: MvccSnapshot[applied={T|T < 7114030778932461568}]
I20250114 03:05:20.657407 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8039
I20250114 03:05:20.664315 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8170
I20250114 03:05:20.670866 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8301
I20250114 03:05:20.677227 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8432
I20250114 03:05:20.683037 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8563
2025-01-14T03:05:20Z chronyd exiting
[ OK ] RaftConsensusElectionITest.ElectPendingVoter (2600 ms)
[ RUN ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy
I20250114 03:05:20.706588 1461 test_util.cc:274] Using random seed: -724253674
I20250114 03:05:20.707376 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:20.707413 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:20.707424 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:05:20.707435 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:05:20.707445 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:20Z Disabled control of system clock
I20250114 03:05:20.715989 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:36005
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:36005
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20250114 03:05:20.824203 8795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:20.824479 8795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:20.824558 8795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:20.827823 8795 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:20.827900 8795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:20.827939 8795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:20.827980 8795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:20.828022 8795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:20.831789 8795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:36005
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:36005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:20.832934 8795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:20.834069 8795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:20.841857 8803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:20.842597 8800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:20.843365 8801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:20.844321 8795 server_base.cc:1034] running on GCE node
I20250114 03:05:20.844898 8795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:20.845820 8795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:20.846990 8795 hybrid_clock.cc:648] HybridClock initialized: now 1736823920846948 us; error 55 us; skew 500 ppm
I20250114 03:05:20.849074 8795 webserver.cc:458] Webserver started at http://127.1.109.126:38309/ using document root <none> and password file <none>
I20250114 03:05:20.849642 8795 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:20.849697 8795 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:20.849880 8795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:20.851588 8795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ebe54eedff084284be4ddb1581930d9e"
format_stamp: "Formatted at 2025-01-14 03:05:20 on dist-test-slave-r16g"
I20250114 03:05:20.852057 8795 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ebe54eedff084284be4ddb1581930d9e"
format_stamp: "Formatted at 2025-01-14 03:05:20 on dist-test-slave-r16g"
I20250114 03:05:20.855772 8795 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250114 03:05:20.858044 8809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:20.859144 8795 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:20.859266 8795 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ebe54eedff084284be4ddb1581930d9e"
format_stamp: "Formatted at 2025-01-14 03:05:20 on dist-test-slave-r16g"
I20250114 03:05:20.859364 8795 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:20.883991 8795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:20.884711 8795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:20.884897 8795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:20.891741 8795 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:36005
I20250114 03:05:20.891752 8861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:36005 every 8 connection(s)
I20250114 03:05:20.892774 8795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:20.895412 8862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:20.900444 8862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Bootstrap starting.
I20250114 03:05:20.901535 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8795
I20250114 03:05:20.901630 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:20.902902 8862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:20.903653 8862 log.cc:826] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:20.905326 8862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: No bootstrap required, opened a new log
I20250114 03:05:20.908480 8862 raft_consensus.cc:357] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:20.908782 8862 raft_consensus.cc:383] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:20.908869 8862 raft_consensus.cc:738] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe54eedff084284be4ddb1581930d9e, State: Initialized, Role: FOLLOWER
I20250114 03:05:20.909330 8862 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:20.909476 8862 raft_consensus.cc:397] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:20.909538 8862 raft_consensus.cc:491] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:20.909761 8862 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:20.910595 8862 raft_consensus.cc:513] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:20.910955 8862 leader_election.cc:304] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe54eedff084284be4ddb1581930d9e; no voters:
I20250114 03:05:20.911201 8862 leader_election.cc:290] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:20.911314 8867 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:20.911521 8867 raft_consensus.cc:695] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 LEADER]: Becoming Leader. State: Replica: ebe54eedff084284be4ddb1581930d9e, State: Running, Role: LEADER
I20250114 03:05:20.911871 8867 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:20.912246 8862 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:20.913524 8869 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe54eedff084284be4ddb1581930d9e. Latest consensus state: current_term: 1 leader_uuid: "ebe54eedff084284be4ddb1581930d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } } }
I20250114 03:05:20.913523 8868 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe54eedff084284be4ddb1581930d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } } }
I20250114 03:05:20.913678 8869 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:20.913678 8868 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: This master's current role is: LEADER
W20250114 03:05:20.917022 8883 catalog_manager.cc:1559] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250114 03:05:20.917083 8883 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250114 03:05:20.917191 8877 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:20.917901 8877 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:20.922933 8877 catalog_manager.cc:1348] Generated new cluster ID: a75a337af34c41af9035930d73833d5f
I20250114 03:05:20.923002 8877 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:20.930037 8877 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:20.930974 8877 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:20.937573 8877 catalog_manager.cc:5899] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Generated new TSK 0
I20250114 03:05:20.938241 8877 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:20.944495 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36005
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
W20250114 03:05:21.054900 8886 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250114 03:05:21.055200 8886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:21.055243 8886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:21.055315 8886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:21.058465 8886 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250114 03:05:21.058526 8886 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250114 03:05:21.058573 8886 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:21.058625 8886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:21.058722 8886 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250114 03:05:21.058770 8886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:21.062659 8886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36005
--never_fsync=true
--heap_profile_path=/tmp/kudu.8886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:21.063839 8886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:21.064992 8886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:21.074070 8892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:21.074911 8891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:21.075650 8894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:21.076613 8886 server_base.cc:1034] running on GCE node
I20250114 03:05:21.077077 8886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:21.077684 8886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:21.078837 8886 hybrid_clock.cc:648] HybridClock initialized: now 1736823921078809 us; error 41 us; skew 500 ppm
I20250114 03:05:21.080902 8886 webserver.cc:458] Webserver started at http://127.1.109.65:45517/ using document root <none> and password file <none>
I20250114 03:05:21.081486 8886 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:21.081539 8886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:21.081696 8886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:21.083336 8886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "62c797d1626f4638990c88ecc0aa2ad8"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.083786 8886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "62c797d1626f4638990c88ecc0aa2ad8"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.087231 8886 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:05:21.089326 8900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.090329 8886 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:21.090428 8886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "62c797d1626f4638990c88ecc0aa2ad8"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.090500 8886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:21.102810 8886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:21.103371 8886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:21.103495 8886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:21.103952 8886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:21.104799 8886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:21.104903 8886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.104947 8886 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:21.104969 8886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.113554 8886 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:38359
I20250114 03:05:21.113606 9013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:38359 every 8 connection(s)
I20250114 03:05:21.114629 8886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:21.120765 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 8886
I20250114 03:05:21.120900 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:21.123314 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36005
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250114 03:05:21.123963 9014 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36005
I20250114 03:05:21.124225 9014 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:21.124863 9014 heartbeater.cc:510] Master 127.1.109.126:36005 requested a full tablet report, sending...
I20250114 03:05:21.126346 8822 ts_manager.cc:194] Registered new tserver with Master: 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359)
I20250114 03:05:21.127539 8822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:37999
W20250114 03:05:21.231017 9017 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250114 03:05:21.231278 9017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:21.231323 9017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:21.231393 9017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:21.234344 9017 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250114 03:05:21.234416 9017 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250114 03:05:21.234463 9017 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:21.234530 9017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:21.234638 9017 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250114 03:05:21.234678 9017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:21.238369 9017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36005
--never_fsync=true
--heap_profile_path=/tmp/kudu.9017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:21.239440 9017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:21.240556 9017 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:21.248850 9023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:21.249616 9022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:21.250530 9025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:21.251324 9017 server_base.cc:1034] running on GCE node
I20250114 03:05:21.251801 9017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:21.252336 9017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:21.253480 9017 hybrid_clock.cc:648] HybridClock initialized: now 1736823921253459 us; error 33 us; skew 500 ppm
I20250114 03:05:21.255313 9017 webserver.cc:458] Webserver started at http://127.1.109.66:46737/ using document root <none> and password file <none>
I20250114 03:05:21.255904 9017 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:21.255964 9017 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:21.256142 9017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:21.257808 9017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "82e12f0b53bc4e4fb39686d5d9684954"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.258265 9017 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "82e12f0b53bc4e4fb39686d5d9684954"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.261682 9017 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250114 03:05:21.263803 9031 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.264801 9017 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250114 03:05:21.264961 9017 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "82e12f0b53bc4e4fb39686d5d9684954"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.265051 9017 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:21.291077 9017 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:21.291750 9017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:21.291908 9017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:21.292528 9017 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:21.293462 9017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:21.293517 9017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.293560 9017 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:21.293592 9017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.302461 9017 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:32897
I20250114 03:05:21.302503 9144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:32897 every 8 connection(s)
I20250114 03:05:21.303380 9017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:21.309573 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9017
I20250114 03:05:21.309703 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:21.311928 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36005
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250114 03:05:21.312111 9145 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36005
I20250114 03:05:21.312316 9145 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:21.312736 9145 heartbeater.cc:510] Master 127.1.109.126:36005 requested a full tablet report, sending...
I20250114 03:05:21.313706 8822 ts_manager.cc:194] Registered new tserver with Master: 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:21.314328 8822 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:34741
W20250114 03:05:21.418296 9148 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250114 03:05:21.418586 9148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:21.418627 9148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:21.418685 9148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:21.421837 9148 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250114 03:05:21.421900 9148 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250114 03:05:21.421945 9148 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:21.422004 9148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:21.422106 9148 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250114 03:05:21.422147 9148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:21.426004 9148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36005
--never_fsync=true
--heap_profile_path=/tmp/kudu.9148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:21.427111 9148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:21.428191 9148 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:21.436451 9156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:21.437075 9154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:21.437924 9153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:21.438805 9148 server_base.cc:1034] running on GCE node
I20250114 03:05:21.439208 9148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:21.439795 9148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:21.440912 9148 hybrid_clock.cc:648] HybridClock initialized: now 1736823921440885 us; error 42 us; skew 500 ppm
I20250114 03:05:21.442832 9148 webserver.cc:458] Webserver started at http://127.1.109.67:40457/ using document root <none> and password file <none>
I20250114 03:05:21.443392 9148 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:21.443449 9148 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:21.443621 9148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:21.445313 9148 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e6648bf13d084e3a9c177fd7d38bc6e6"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.445772 9148 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e6648bf13d084e3a9c177fd7d38bc6e6"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.449091 9148 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20250114 03:05:21.451220 9162 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.452241 9148 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:21.452349 9148 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e6648bf13d084e3a9c177fd7d38bc6e6"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:21.452433 9148 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:21.467720 9148 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:21.468418 9148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:21.468587 9148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:21.469259 9148 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:21.470178 9148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:21.470228 9148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.470302 9148 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:21.470328 9148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:21.479089 9148 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:46041
I20250114 03:05:21.479122 9275 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:46041 every 8 connection(s)
I20250114 03:05:21.480070 9148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:21.488109 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9148
I20250114 03:05:21.488245 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:21.489514 9276 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36005
I20250114 03:05:21.489769 9276 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:21.490334 9276 heartbeater.cc:510] Master 127.1.109.126:36005 requested a full tablet report, sending...
I20250114 03:05:21.491267 8826 ts_manager.cc:194] Registered new tserver with Master: e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041)
I20250114 03:05:21.491823 8826 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:45241
I20250114 03:05:21.502229 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:21.510901 8826 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34776:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250114 03:05:21.529260 8948 tablet_service.cc:1467] Processing CreateTablet for tablet a8c2353cd1d84f0c92405d73bd142a1a (DEFAULT_TABLE table=TestTable [id=346dea02923840b2a7b32172df0a3f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:21.529268 9209 tablet_service.cc:1467] Processing CreateTablet for tablet a8c2353cd1d84f0c92405d73bd142a1a (DEFAULT_TABLE table=TestTable [id=346dea02923840b2a7b32172df0a3f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:21.529268 9079 tablet_service.cc:1467] Processing CreateTablet for tablet a8c2353cd1d84f0c92405d73bd142a1a (DEFAULT_TABLE table=TestTable [id=346dea02923840b2a7b32172df0a3f49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:21.530202 8948 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2353cd1d84f0c92405d73bd142a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:21.530206 9209 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2353cd1d84f0c92405d73bd142a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:21.530206 9079 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2353cd1d84f0c92405d73bd142a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:21.535310 9293 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Bootstrap starting.
I20250114 03:05:21.536623 9294 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Bootstrap starting.
I20250114 03:05:21.536648 9295 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Bootstrap starting.
I20250114 03:05:21.537906 9293 tablet_bootstrap.cc:654] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:21.538565 9295 tablet_bootstrap.cc:654] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:21.538672 9294 tablet_bootstrap.cc:654] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:21.538750 9293 log.cc:826] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:21.539278 9295 log.cc:826] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:21.539409 9294 log.cc:826] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:21.540665 9293 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: No bootstrap required, opened a new log
I20250114 03:05:21.540794 9295 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: No bootstrap required, opened a new log
I20250114 03:05:21.540881 9294 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: No bootstrap required, opened a new log
I20250114 03:05:21.540897 9293 ts_tablet_manager.cc:1397] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20250114 03:05:21.540949 9295 ts_tablet_manager.cc:1397] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.001s
I20250114 03:05:21.541052 9294 ts_tablet_manager.cc:1397] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:21.544144 9294 raft_consensus.cc:357] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.544335 9294 raft_consensus.cc:383] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:21.544412 9294 raft_consensus.cc:738] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e12f0b53bc4e4fb39686d5d9684954, State: Initialized, Role: FOLLOWER
I20250114 03:05:21.544689 9295 raft_consensus.cc:357] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.544689 9293 raft_consensus.cc:357] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.544926 9293 raft_consensus.cc:383] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:21.544916 9295 raft_consensus.cc:383] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:21.544903 9294 consensus_queue.cc:260] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.544981 9295 raft_consensus.cc:738] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6648bf13d084e3a9c177fd7d38bc6e6, State: Initialized, Role: FOLLOWER
I20250114 03:05:21.545017 9293 raft_consensus.cc:738] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c797d1626f4638990c88ecc0aa2ad8, State: Initialized, Role: FOLLOWER
I20250114 03:05:21.545413 9295 consensus_queue.cc:260] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.545413 9293 consensus_queue.cc:260] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.545825 9294 ts_tablet_manager.cc:1428] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:21.546149 9014 heartbeater.cc:502] Master 127.1.109.126:36005 was elected leader, sending a full tablet report...
I20250114 03:05:21.546281 9293 ts_tablet_manager.cc:1428] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:21.546281 9295 ts_tablet_manager.cc:1428] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20250114 03:05:21.546622 9276 heartbeater.cc:502] Master 127.1.109.126:36005 was elected leader, sending a full tablet report...
I20250114 03:05:21.546720 9145 heartbeater.cc:502] Master 127.1.109.126:36005 was elected leader, sending a full tablet report...
W20250114 03:05:21.580988 9277 tablet.cc:2367] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:05:21.604848 9146 tablet.cc:2367] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250114 03:05:21.616294 9015 tablet.cc:2367] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:21.632220 9299 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:05:21.632417 9299 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.633599 9299 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041)
I20250114 03:05:21.637835 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "82e12f0b53bc4e4fb39686d5d9684954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c797d1626f4638990c88ecc0aa2ad8" is_pre_election: true
I20250114 03:05:21.637840 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "82e12f0b53bc4e4fb39686d5d9684954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" is_pre_election: true
I20250114 03:05:21.638131 9230 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e12f0b53bc4e4fb39686d5d9684954 in term 0.
I20250114 03:05:21.638136 8968 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82e12f0b53bc4e4fb39686d5d9684954 in term 0.
I20250114 03:05:21.638585 9033 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8, 82e12f0b53bc4e4fb39686d5d9684954; no voters:
I20250114 03:05:21.638831 9299 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:05:21.638900 9299 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:05:21.638931 9299 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:21.639845 9299 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.640199 9299 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [CANDIDATE]: Term 1 election: Requested vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041)
I20250114 03:05:21.640563 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "82e12f0b53bc4e4fb39686d5d9684954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c797d1626f4638990c88ecc0aa2ad8"
I20250114 03:05:21.640574 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "82e12f0b53bc4e4fb39686d5d9684954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6"
I20250114 03:05:21.640667 8968 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:21.640707 9230 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:21.641548 9230 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82e12f0b53bc4e4fb39686d5d9684954 in term 1.
I20250114 03:05:21.641583 8968 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82e12f0b53bc4e4fb39686d5d9684954 in term 1.
I20250114 03:05:21.641865 9033 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8, 82e12f0b53bc4e4fb39686d5d9684954; no voters:
I20250114 03:05:21.642052 9299 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:21.642238 9299 raft_consensus.cc:695] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 1 LEADER]: Becoming Leader. State: Replica: 82e12f0b53bc4e4fb39686d5d9684954, State: Running, Role: LEADER
I20250114 03:05:21.642524 9299 consensus_queue.cc:237] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:21.645431 8825 catalog_manager.cc:5526] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66). New cstate: current_term: 1 leader_uuid: "82e12f0b53bc4e4fb39686d5d9684954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } } }
I20250114 03:05:21.680773 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:21.682266 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 62c797d1626f4638990c88ecc0aa2ad8 to finish bootstrapping
I20250114 03:05:21.687589 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 82e12f0b53bc4e4fb39686d5d9684954 to finish bootstrapping
I20250114 03:05:21.692588 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e6648bf13d084e3a9c177fd7d38bc6e6 to finish bootstrapping
I20250114 03:05:21.712047 1461 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0 0.0
I20250114 03:05:21.814893 1461 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0 0.0
I20250114 03:05:22.017760 1461 cluster_itest_util.cc:256] Not converged past 1 yet: 1.1 0.0 0.0
I20250114 03:05:22.125546 9299 consensus_queue.cc:1035] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:22.130885 9299 consensus_queue.cc:1035] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:22.433794 1461 test_util.cc:274] Using random seed: -722526474
I20250114 03:05:22.739745 1461 test_util.cc:274] Using random seed: -722220523
I20250114 03:05:23.144699 1461 test_util.cc:274] Using random seed: -721815584
I20250114 03:05:23.630931 9314 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82e12f0b53bc4e4fb39686d5d9684954)
I20250114 03:05:23.631090 9314 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:23.632233 9314 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:23.632526 9315 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82e12f0b53bc4e4fb39686d5d9684954)
I20250114 03:05:23.632675 9315 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:23.633664 9315 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:23.636374 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62c797d1626f4638990c88ecc0aa2ad8" is_pre_election: true
I20250114 03:05:23.636509 8968 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6648bf13d084e3a9c177fd7d38bc6e6 in term 1.
I20250114 03:05:23.636888 9164 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8, e6648bf13d084e3a9c177fd7d38bc6e6; no voters:
I20250114 03:05:23.637089 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "62c797d1626f4638990c88ecc0aa2ad8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" is_pre_election: true
I20250114 03:05:23.637104 9314 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 03:05:23.637224 9230 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 62c797d1626f4638990c88ecc0aa2ad8 in term 1.
I20250114 03:05:23.637282 9314 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82e12f0b53bc4e4fb39686d5d9684954)
I20250114 03:05:23.637342 9314 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:23.637552 8901 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8, e6648bf13d084e3a9c177fd7d38bc6e6; no voters:
I20250114 03:05:23.637749 9315 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250114 03:05:23.637804 9315 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 82e12f0b53bc4e4fb39686d5d9684954)
I20250114 03:05:23.637833 9315 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:23.638302 9314 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:23.638659 9314 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 2 election: Requested vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:23.638653 9315 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:23.638939 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62c797d1626f4638990c88ecc0aa2ad8"
I20250114 03:05:23.639003 9315 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [CANDIDATE]: Term 2 election: Requested vote from peers e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:23.639079 8968 raft_consensus.cc:2388] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e6648bf13d084e3a9c177fd7d38bc6e6 in current term 2: Already voted for candidate 62c797d1626f4638990c88ecc0aa2ad8 in this term.
I20250114 03:05:23.639281 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "62c797d1626f4638990c88ecc0aa2ad8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6"
I20250114 03:05:23.639425 9230 raft_consensus.cc:2388] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 62c797d1626f4638990c88ecc0aa2ad8 in current term 2: Already voted for candidate e6648bf13d084e3a9c177fd7d38bc6e6 in this term.
I20250114 03:05:23.649645 1461 test_util.cc:274] Using random seed: -721310620
I20250114 03:05:24.254756 1461 test_util.cc:274] Using random seed: -720705511
I20250114 03:05:24.959738 1461 test_util.cc:274] Using random seed: -720000530
W20250114 03:05:25.481309 8903 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.1.109.66:32897 for RPC RequestConsensusVote timed out after 1.846s (ON_OUTBOUND_QUEUE)
W20250114 03:05:25.491731 9165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.1.109.66:32897 for RPC RequestConsensusVote timed out after 1.852s (ON_OUTBOUND_QUEUE)
I20250114 03:05:25.764926 1461 test_util.cc:274] Using random seed: -719195342
I20250114 03:05:26.670178 1461 test_util.cc:274] Using random seed: -718290089
W20250114 03:05:26.845599 9316 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:05:23.632500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.109.66:32897 (local address 127.1.109.67:41657)
0114 03:05:23.632893 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0114 03:05:23.632911 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0114 03:05:23.633025 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0114 03:05:26.845088 (+3212063us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":71,"threads_started":1}
W20250114 03:05:26.845625 9320 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:05:23.633853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.109.66:32897 (local address 127.1.109.65:40791)
0114 03:05:23.634157 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0114 03:05:23.634170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0114 03:05:23.634250 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0114 03:05:26.845087 (+3210837us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":53,"threads_started":1}
W20250114 03:05:26.845837 9165 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 4 requested bytes
W20250114 03:05:26.845853 8903 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 4 requested bytes
I20250114 03:05:26.845952 8903 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8; no voters: 82e12f0b53bc4e4fb39686d5d9684954, e6648bf13d084e3a9c177fd7d38bc6e6
W20250114 03:05:26.846150 9165 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 4 requested bytes
I20250114 03:05:26.846215 9165 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e6648bf13d084e3a9c177fd7d38bc6e6; no voters: 62c797d1626f4638990c88ecc0aa2ad8, 82e12f0b53bc4e4fb39686d5d9684954
I20250114 03:05:26.846278 9324 raft_consensus.cc:2743] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20250114 03:05:26.846305 8903 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 4 requested bytes
I20250114 03:05:26.846482 9325 raft_consensus.cc:2743] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20250114 03:05:27.675967 1461 test_util.cc:274] Using random seed: -717284303
I20250114 03:05:28.781536 1461 test_util.cc:274] Using random seed: -716178732
I20250114 03:05:29.025573 9326 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:05:29.025741 9326 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:29.026244 9326 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:29.026831 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62c797d1626f4638990c88ecc0aa2ad8" is_pre_election: true
I20250114 03:05:29.027012 8968 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6648bf13d084e3a9c177fd7d38bc6e6 in term 2.
I20250114 03:05:29.027417 9164 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8, e6648bf13d084e3a9c177fd7d38bc6e6; no voters:
I20250114 03:05:29.027635 9326 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250114 03:05:29.027698 9326 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:05:29.027769 9326 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:05:29.028707 9326 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:29.029098 9326 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 3 election: Requested vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:29.029461 8968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "62c797d1626f4638990c88ecc0aa2ad8"
I20250114 03:05:29.029603 8968 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 2 FOLLOWER]: Advancing to term 3
I20250114 03:05:29.030390 8968 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6648bf13d084e3a9c177fd7d38bc6e6 in term 3.
I20250114 03:05:29.030678 9164 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c797d1626f4638990c88ecc0aa2ad8, e6648bf13d084e3a9c177fd7d38bc6e6; no voters:
I20250114 03:05:29.030864 9326 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Leader election won for term 3
I20250114 03:05:29.031054 9326 raft_consensus.cc:695] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 LEADER]: Becoming Leader. State: Replica: e6648bf13d084e3a9c177fd7d38bc6e6, State: Running, Role: LEADER
I20250114 03:05:29.031347 9326 consensus_queue.cc:237] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
I20250114 03:05:29.033612 8825 catalog_manager.cc:5526] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 reported cstate change: term changed from 1 to 3, leader changed from 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) to e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67). New cstate: current_term: 3 leader_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } } }
I20250114 03:05:29.603104 8968 raft_consensus.cc:1270] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Refusing update from remote peer e6648bf13d084e3a9c177fd7d38bc6e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 03:05:29.603817 9334 consensus_queue.cc:1035] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:05:29.887390 1461 test_util.cc:274] Using random seed: -715072877
I20250114 03:05:29.902341 1461 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250114 03:05:29.902629 1461 raft_consensus-itest-base.cc:268] Waiting until we've written at least 4MB...
I20250114 03:05:29.999074 1461 raft_consensus-itest-base.cc:274] Waiting for log GC on e6648bf13d084e3a9c177fd7d38bc6e6
I20250114 03:05:30.010696 9277 maintenance_manager.cc:418] P e6648bf13d084e3a9c177fd7d38bc6e6: Scheduling FlushMRSOp(a8c2353cd1d84f0c92405d73bd142a1a): 4602350 bytes log retention, and flush 6119689 bytes memory
I20250114 03:05:30.047199 9015 maintenance_manager.cc:418] P 62c797d1626f4638990c88ecc0aa2ad8: Scheduling FlushMRSOp(a8c2353cd1d84f0c92405d73bd142a1a): 4470585 bytes log retention, and flush 6119689 bytes memory
I20250114 03:05:30.060032 9167 maintenance_manager.cc:642] P e6648bf13d084e3a9c177fd7d38bc6e6: FlushMRSOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.049s user 0.047s sys 0.000s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":266,"dirs.queue_time_us":4179,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":40,"thread_start_us":248,"threads_started":2}
I20250114 03:05:30.061084 9277 maintenance_manager.cc:418] P e6648bf13d084e3a9c177fd7d38bc6e6: Scheduling LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a): free 2761400 bytes of WAL
I20250114 03:05:30.061419 9167 log_reader.cc:385] T a8c2353cd1d84f0c92405d73bd142a1a: removed 3 log segments from log reader
I20250114 03:05:30.061553 9167 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000001 (ops 1-9)
I20250114 03:05:30.061709 9167 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000002 (ops 10-16)
I20250114 03:05:30.061822 9167 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000003 (ops 17-23)
I20250114 03:05:30.062892 9167 maintenance_manager.cc:642] P e6648bf13d084e3a9c177fd7d38bc6e6: LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20250114 03:05:30.063217 9277 maintenance_manager.cc:418] P e6648bf13d084e3a9c177fd7d38bc6e6: Scheduling UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a): 579 bytes on disk
I20250114 03:05:30.063900 9167 maintenance_manager.cc:642] P e6648bf13d084e3a9c177fd7d38bc6e6: UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20250114 03:05:30.067076 1461 raft_consensus-itest-base.cc:279] Log GC complete on e6648bf13d084e3a9c177fd7d38bc6e6
I20250114 03:05:30.116185 8905 maintenance_manager.cc:642] P 62c797d1626f4638990c88ecc0aa2ad8: FlushMRSOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.068s user 0.060s sys 0.004s Metrics: {"bytes_written":136001,"cfile_init":1,"compiler_manager_pool.queue_time_us":289,"dirs.queue_time_us":208,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":950,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":687,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":40,"thread_start_us":252,"threads_started":2}
I20250114 03:05:30.117180 9015 maintenance_manager.cc:418] P 62c797d1626f4638990c88ecc0aa2ad8: Scheduling LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a): free 2761256 bytes of WAL
I20250114 03:05:30.117524 8905 log_reader.cc:385] T a8c2353cd1d84f0c92405d73bd142a1a: removed 3 log segments from log reader
I20250114 03:05:30.117628 8905 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000001 (ops 1-9)
I20250114 03:05:30.117764 8905 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000002 (ops 10-16)
I20250114 03:05:30.117929 8905 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000003 (ops 17-23)
I20250114 03:05:30.118960 8905 maintenance_manager.cc:642] P 62c797d1626f4638990c88ecc0aa2ad8: LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20250114 03:05:30.119305 9015 maintenance_manager.cc:418] P 62c797d1626f4638990c88ecc0aa2ad8: Scheduling UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a): 579 bytes on disk
I20250114 03:05:30.119976 8905 maintenance_manager.cc:642] P 62c797d1626f4638990c88ecc0aa2ad8: UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20250114 03:05:31.364285 9165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.1.109.66:32897 for RPC RequestConsensusVote timed out after 2.336s (ON_OUTBOUND_QUEUE) [suppressed 1 similar messages]
I20250114 03:05:32.068466 1461 raft_consensus-itest-base.cc:295] Servers converged with original term 3
I20250114 03:05:32.068547 1461 raft_consensus-itest-base.cc:300] Resuming 82e12f0b53bc4e4fb39686d5d9684954
I20250114 03:05:32.072017 8968 raft_consensus.cc:1235] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Rejecting Update request from peer 82e12f0b53bc4e4fb39686d5d9684954 for earlier term 1. Current term is 3. Ops: []
I20250114 03:05:32.073021 9230 raft_consensus.cc:1235] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 LEADER]: Rejecting Update request from peer 82e12f0b53bc4e4fb39686d5d9684954 for earlier term 1. Current term is 3. Ops: []
I20250114 03:05:32.073685 9299 consensus_queue.cc:1046] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 42, Time since last communication: 0.000s
W20250114 03:05:32.073856 9327 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:05:29.026551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.1.109.66:32897 (local address 127.1.109.67:50585)
0114 03:05:29.026813 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0114 03:05:29.026831 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0114 03:05:29.026939 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0114 03:05:32.073745 (+3046806us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 25 requested bytes
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":79,"threads_started":1}
I20250114 03:05:32.073863 9299 consensus_queue.cc:1225] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [LEADER]: Peer e6648bf13d084e3a9c177fd7d38bc6e6 log is divergent from this leader: its last log entry 3.42 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 42
I20250114 03:05:32.073930 9299 raft_consensus.cc:3049] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:32.073962 9299 raft_consensus.cc:738] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 82e12f0b53bc4e4fb39686d5d9684954, State: Running, Role: LEADER
W20250114 03:05:32.073995 9165 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 25 requested bytes
I20250114 03:05:32.074093 9299 consensus_queue.cc:260] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } }
W20250114 03:05:32.074121 9165 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Timed out: Client connection negotiation failed: client connection to 127.1.109.66:32897: received 0 of 25 requested bytes
I20250114 03:05:32.074350 9299 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 1 FOLLOWER]: Advancing to term 3
W20250114 03:05:32.075793 9372 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:05:32.072502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.109.67:50585 (local address 127.1.109.66:32897)
0114 03:05:32.073474 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0114 03:05:32.073479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0114 03:05:32.073493 (+ 14us) server_negotiation.cc:407] Connection header received
0114 03:05:32.073540 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0114 03:05:32.073543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0114 03:05:32.073589 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0114 03:05:32.075175 (+ 1586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.109.67:50585: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":62,"threads_started":1}
W20250114 03:05:32.075784 9371 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:05:32.072643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.109.65:40791 (local address 127.1.109.66:32897)
0114 03:05:32.075285 (+ 2642us) server_negotiation.cc:206] Beginning negotiation
0114 03:05:32.075289 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0114 03:05:32.075300 (+ 11us) server_negotiation.cc:407] Connection header received
0114 03:05:32.075338 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0114 03:05:32.075340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0114 03:05:32.075377 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0114 03:05:32.075488 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.109.65:40791: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":2582,"thread_start_us":45,"threads_started":1}
W20250114 03:05:32.076704 9373 negotiation.cc:337] Failed RPC negotiation. Trace:
0114 03:05:32.074736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.1.109.67:41657 (local address 127.1.109.66:32897)
0114 03:05:32.075912 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0114 03:05:32.075916 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0114 03:05:32.075932 (+ 16us) server_negotiation.cc:407] Connection header received
0114 03:05:32.075978 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0114 03:05:32.075980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0114 03:05:32.076023 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0114 03:05:32.076136 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.1.109.67:41657: BlockingWrite error: write error: Broken pipe (error 32)
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":57,"threads_started":1}
I20250114 03:05:32.079358 9098 raft_consensus.cc:1270] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: Refusing update from remote peer e6648bf13d084e3a9c177fd7d38bc6e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20250114 03:05:32.079857 9369 consensus_queue.cc:1035] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250114 03:05:32.080173 9369 consensus_queue.cc:740] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: The logs necessary to catch up peer 82e12f0b53bc4e4fb39686d5d9684954 have been garbage collected. The follower will never be able to catch up (Not found: failed to read ops 2..35: Segment 1 which contained index 2 has been GCed)
I20250114 03:05:32.080492 9369 raft_consensus.cc:1007] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Attempting to remove follower 82e12f0b53bc4e4fb39686d5d9684954 from the Raft config. Reason: The logs necessary to catch up peer 82e12f0b53bc4e4fb39686d5d9684954 have been garbage collected. The replica will never be able to catch up
I20250114 03:05:32.081156 9369 consensus_queue.cc:237] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 42, Committed index: 42, Last appended: 3.42, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } }
I20250114 03:05:32.082275 8968 raft_consensus.cc:1270] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Refusing update from remote peer e6648bf13d084e3a9c177fd7d38bc6e6: Log matching property violated. Preceding OpId in replica: term: 3 index: 42. Preceding OpId from leader: term: 3 index: 43. (index mismatch)
I20250114 03:05:32.082675 9375 consensus_queue.cc:1035] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 43, Last known committed idx: 42, Time since last communication: 0.000s
I20250114 03:05:32.084419 9369 raft_consensus.cc:2949] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 LEADER]: Committing config change with OpId 3.43: config changed from index -1 to 43, VOTER 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } }
I20250114 03:05:32.084852 8968 raft_consensus.cc:2949] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Committing config change with OpId 3.43: config changed from index -1 to 43, VOTER 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) evicted. New config: { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } }
I20250114 03:05:32.086810 8825 catalog_manager.cc:5526] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 reported cstate change: config changed from index -1 to 43, VOTER 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) evicted. New cstate: current_term: 3 leader_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" committed_config { opid_index: 43 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } }
I20250114 03:05:32.090227 9079 tablet_service.cc:1514] Processing DeleteTablet for tablet a8c2353cd1d84f0c92405d73bd142a1a with delete_type TABLET_DATA_TOMBSTONED (TS 82e12f0b53bc4e4fb39686d5d9684954 not found in new config with opid_index 43) from {username='slave'} at 127.0.0.1:36618
I20250114 03:05:32.090750 9383 tablet_replica.cc:331] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: stopping tablet replica
I20250114 03:05:32.091029 9383 raft_consensus.cc:2238] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:32.091233 9383 raft_consensus.cc:2267] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:32.092105 9383 ts_tablet_manager.cc:1905] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:32.094084 9383 ts_tablet_manager.cc:1918] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:32.094172 9383 log.cc:1198] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a
I20250114 03:05:32.094885 8812 catalog_manager.cc:4872] TS 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): tablet a8c2353cd1d84f0c92405d73bd142a1a (table TestTable [id=346dea02923840b2a7b32172df0a3f49]) successfully deleted
I20250114 03:05:32.095314 9139 generic_service.cc:175] {username='slave'} at 127.0.0.1:36630 changed flags via RPC: tablet_copy_fault_crash_on_fetch_all from '0' to '1.0'
I20250114 03:05:32.096524 9230 consensus_queue.cc:237] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 3.43, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:32.097826 8968 raft_consensus.cc:1270] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Refusing update from remote peer e6648bf13d084e3a9c177fd7d38bc6e6: Log matching property violated. Preceding OpId in replica: term: 3 index: 43. Preceding OpId from leader: term: 3 index: 44. (index mismatch)
W20250114 03:05:32.098191 9165 consensus_peers.cc:487] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 -> Peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Couldn't send request to peer 82e12f0b53bc4e4fb39686d5d9684954. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250114 03:05:32.098233 9375 consensus_queue.cc:1035] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 44, Last known committed idx: 43, Time since last communication: 0.000s
I20250114 03:05:32.099861 9369 raft_consensus.cc:2949] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 LEADER]: Committing config change with OpId 3.44: config changed from index 43 to 44, VOTER 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } } }
I20250114 03:05:32.100347 8968 raft_consensus.cc:2949] T a8c2353cd1d84f0c92405d73bd142a1a P 62c797d1626f4638990c88ecc0aa2ad8 [term 3 FOLLOWER]: Committing config change with OpId 3.44: config changed from index 43 to 44, VOTER 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) added. New config: { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } } }
I20250114 03:05:32.102232 8824 catalog_manager.cc:5526] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 reported cstate change: config changed from index 43 to 44, VOTER 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66) added. New cstate: current_term: 3 leader_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } } }
I20250114 03:05:32.515780 9384 ts_tablet_manager.cc:927] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Initiating tablet copy from peer e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041)
I20250114 03:05:32.516438 9384 tablet_copy_client.cc:323] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet copy: Beginning tablet copy session from remote peer at address 127.1.109.67:46041
I20250114 03:05:32.517246 9250 tablet_copy_service.cc:140] P e6648bf13d084e3a9c177fd7d38bc6e6: Received BeginTabletCopySession request for tablet a8c2353cd1d84f0c92405d73bd142a1a from peer 82e12f0b53bc4e4fb39686d5d9684954 ({username='slave'} at 127.1.109.66:33445)
I20250114 03:05:32.517406 9250 tablet_copy_service.cc:161] P e6648bf13d084e3a9c177fd7d38bc6e6: Beginning new tablet copy session on tablet a8c2353cd1d84f0c92405d73bd142a1a from peer 82e12f0b53bc4e4fb39686d5d9684954 at {username='slave'} at 127.1.109.66:33445: session id = 82e12f0b53bc4e4fb39686d5d9684954-a8c2353cd1d84f0c92405d73bd142a1a
I20250114 03:05:32.518929 9250 tablet_copy_source_session.cc:215] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Tablet Copy: opened 5 blocks and 3 log segments
I20250114 03:05:32.520452 9384 ts_tablet_manager.cc:1905] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250114 03:05:32.522437 9384 ts_tablet_manager.cc:1918] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20250114 03:05:32.522622 9384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2353cd1d84f0c92405d73bd142a1a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250114 03:05:32.523599 9384 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20250114 03:05:32.523635 9384 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20250114 03:05:32.523661 9384 fault_injection.cc:59] injecting fault for kudu (pid 9017): FLAGS_tablet_copy_fault_crash_on_fetch_all (process will exit)
I20250114 03:05:32.552989 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8886
I20250114 03:05:32.560518 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 9148
I20250114 03:05:32.567685 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 8795
I20250114 03:05:32.575021 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:36005
--webserver_interface=127.1.109.126
--webserver_port=38309
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:36005
--master_add_server_when_underreplicated=false
--raft_prepare_replacement_before_eviction=false with env {}
W20250114 03:05:32.683905 9386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:32.684181 9386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:32.684253 9386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:32.687481 9386 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:32.687587 9386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:32.687623 9386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:32.687666 9386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:32.687708 9386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:32.691624 9386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--master_add_server_when_underreplicated=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:36005
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:36005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=38309
--never_fsync=true
--heap_profile_path=/tmp/kudu.9386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:32.692777 9386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:32.693961 9386 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250114 03:05:32.702016 9386 server_base.cc:1034] running on GCE node
W20250114 03:05:32.702237 9394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:32.703022 9391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:32.703617 9392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:32.704178 9386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:32.705174 9386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:32.706323 9386 hybrid_clock.cc:648] HybridClock initialized: now 1736823932706308 us; error 28 us; skew 500 ppm
I20250114 03:05:32.708330 9386 webserver.cc:458] Webserver started at http://127.1.109.126:38309/ using document root <none> and password file <none>
I20250114 03:05:32.708948 9386 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:32.709025 9386 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:32.712452 9386 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:32.714500 9400 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:32.715504 9386 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:32.715636 9386 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ebe54eedff084284be4ddb1581930d9e"
format_stamp: "Formatted at 2025-01-14 03:05:20 on dist-test-slave-r16g"
I20250114 03:05:32.716060 9386 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:32.732416 9386 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:32.733103 9386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:32.733268 9386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:32.739765 9386 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:36005
I20250114 03:05:32.739791 9452 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:36005 every 8 connection(s)
I20250114 03:05:32.740901 9386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:32.741075 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9386
I20250114 03:05:32.741585 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:46041
--local_ip_for_outbound_sockets=127.1.109.67
--tserver_master_addrs=127.1.109.126:36005
--webserver_port=40457
--webserver_interface=127.1.109.67
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250114 03:05:32.745268 9453 sys_catalog.cc:263] Verifying existing consensus state
I20250114 03:05:32.746779 9453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Bootstrap starting.
I20250114 03:05:32.760510 9453 log.cc:826] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:32.769069 9453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=5 ignored=0} mutations{seen=5 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250114 03:05:32.769426 9453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Bootstrap complete.
I20250114 03:05:32.773058 9453 raft_consensus.cc:357] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:32.773401 9453 raft_consensus.cc:738] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe54eedff084284be4ddb1581930d9e, State: Initialized, Role: FOLLOWER
I20250114 03:05:32.773862 9453 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:32.773972 9453 raft_consensus.cc:397] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:32.774035 9453 raft_consensus.cc:491] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:32.774138 9453 raft_consensus.cc:3054] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:32.775668 9453 raft_consensus.cc:513] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:32.775972 9453 leader_election.cc:304] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe54eedff084284be4ddb1581930d9e; no voters:
I20250114 03:05:32.776259 9453 leader_election.cc:290] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20250114 03:05:32.776360 9457 raft_consensus.cc:2798] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 2 FOLLOWER]: Leader election won for term 2
I20250114 03:05:32.776541 9457 raft_consensus.cc:695] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [term 2 LEADER]: Becoming Leader. State: Replica: ebe54eedff084284be4ddb1581930d9e, State: Running, Role: LEADER
I20250114 03:05:32.776939 9457 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } }
I20250114 03:05:32.777400 9453 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:32.777865 9457 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe54eedff084284be4ddb1581930d9e. Latest consensus state: current_term: 2 leader_uuid: "ebe54eedff084284be4ddb1581930d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } } }
I20250114 03:05:32.777969 9457 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:32.778352 9458 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ebe54eedff084284be4ddb1581930d9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe54eedff084284be4ddb1581930d9e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36005 } } }
I20250114 03:05:32.778512 9458 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:32.778421 9461 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:32.782207 9461 catalog_manager.cc:670] Loaded metadata for table TestTable [id=346dea02923840b2a7b32172df0a3f49]
I20250114 03:05:32.783579 9461 tablet_loader.cc:96] loaded metadata for tablet a8c2353cd1d84f0c92405d73bd142a1a (table TestTable [id=346dea02923840b2a7b32172df0a3f49])
I20250114 03:05:32.783790 9461 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:32.784451 9461 catalog_manager.cc:1260] Loaded cluster ID: a75a337af34c41af9035930d73833d5f
I20250114 03:05:32.784510 9461 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:32.787184 9461 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:32.788089 9461 catalog_manager.cc:5910] T 00000000000000000000000000000000 P ebe54eedff084284be4ddb1581930d9e: Loaded TSK: 0
I20250114 03:05:32.789335 9461 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250114 03:05:32.856523 9455 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250114 03:05:32.856801 9455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:32.856858 9455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:32.856942 9455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:32.860245 9455 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250114 03:05:32.860311 9455 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250114 03:05:32.860342 9455 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:32.860374 9455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:32.860486 9455 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250114 03:05:32.860522 9455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:32.864495 9455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:46041
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=40457
--tserver_master_addrs=127.1.109.126:36005
--never_fsync=true
--heap_profile_path=/tmp/kudu.9455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:32.865691 9455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:32.866912 9455 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:32.876026 9479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:32.876329 9480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:32.876961 9482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:32.877867 9455 server_base.cc:1034] running on GCE node
I20250114 03:05:32.878259 9455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:32.878868 9455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:32.880033 9455 hybrid_clock.cc:648] HybridClock initialized: now 1736823932880021 us; error 30 us; skew 500 ppm
I20250114 03:05:32.882012 9455 webserver.cc:458] Webserver started at http://127.1.109.67:40457/ using document root <none> and password file <none>
I20250114 03:05:32.882591 9455 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:32.882660 9455 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:32.886211 9455 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20250114 03:05:32.891486 9493 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:32.892481 9455 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250114 03:05:32.892597 9455 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e6648bf13d084e3a9c177fd7d38bc6e6"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:32.893075 9455 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 5
Total live bytes: 136580
Total live bytes (after alignment): 155648
Total number of LBM containers: 6 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:32.903980 9455 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:32.904636 9455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:32.904790 9455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:32.905376 9455 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:32.906539 9500 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20250114 03:05:32.910542 9455 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250114 03:05:32.910600 9455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20250114 03:05:32.910656 9455 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250114 03:05:32.912137 9455 ts_tablet_manager.cc:610] Registered 1 tablets
I20250114 03:05:32.912191 9455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20250114 03:05:32.912460 9500 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Bootstrap starting.
I20250114 03:05:32.922497 9455 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:46041
I20250114 03:05:32.922717 9607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:46041 every 8 connection(s)
I20250114 03:05:32.923746 9455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:32.926386 9500 log.cc:826] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:32.927493 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9455
I20250114 03:05:32.927974 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:32897
--local_ip_for_outbound_sockets=127.1.109.66
--tserver_master_addrs=127.1.109.126:36005
--webserver_port=46737
--webserver_interface=127.1.109.66
--builtin_ntp_servers=127.1.109.84:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=false
--log_compression_codec=no_compression
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--log_async_preallocate_segments=false
--log_min_segments_to_retain=1
--log_max_segments_to_retain=3
--maintenance_manager_polling_interval_ms=100
--log_target_replay_size_mb=1
--max_cell_size_bytes=1000000 with env {}
I20250114 03:05:32.937747 9500 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=6} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 1 replicates
I20250114 03:05:32.940492 9608 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36005
I20250114 03:05:32.940768 9608 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:32.941331 9608 heartbeater.cc:510] Master 127.1.109.126:36005 requested a full tablet report, sending...
I20250114 03:05:32.943514 9500 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=13} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 1 replicates
I20250114 03:05:32.943994 9417 ts_manager.cc:194] Registered new tserver with Master: e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041)
I20250114 03:05:32.946786 9417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:49955
I20250114 03:05:32.948547 9500 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Bootstrap replayed 3/3 log segments. Stats: ops{read=21 overwritten=0 applied=21 ignored=19} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=2. Pending: 0 replicates
I20250114 03:05:32.949133 9500 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Bootstrap complete.
I20250114 03:05:32.950139 9500 ts_tablet_manager.cc:1397] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Time spent bootstrapping tablet: real 0.038s user 0.026s sys 0.010s
I20250114 03:05:32.952142 9500 raft_consensus.cc:357] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:32.952514 9500 raft_consensus.cc:738] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6648bf13d084e3a9c177fd7d38bc6e6, State: Initialized, Role: FOLLOWER
I20250114 03:05:32.952988 9500 consensus_queue.cc:260] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 3.44, Last appended by leader: 44, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:32.953751 9500 ts_tablet_manager.cc:1428] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20250114 03:05:32.954123 9608 heartbeater.cc:502] Master 127.1.109.126:36005 was elected leader, sending a full tablet report...
I20250114 03:05:33.025665 9609 maintenance_manager.cc:418] P e6648bf13d084e3a9c177fd7d38bc6e6: Scheduling UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a): 579 bytes on disk
I20250114 03:05:33.026772 9498 maintenance_manager.cc:642] P e6648bf13d084e3a9c177fd7d38bc6e6: UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20250114 03:05:33.045261 9613 flags.cc:425] Enabled unsafe flag: --max_cell_size_bytes=1000000
W20250114 03:05:33.045495 9613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:33.045516 9613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:33.045558 9613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:33.048947 9613 flags.cc:425] Enabled experimental flag: --log_compression_codec=no_compression
W20250114 03:05:33.049001 9613 flags.cc:425] Enabled experimental flag: --log_max_segments_to_retain=3
W20250114 03:05:33.049036 9613 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=false
W20250114 03:05:33.049065 9613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:33.049136 9613 flags.cc:425] Enabled experimental flag: --log_target_replay_size_mb=1
W20250114 03:05:33.049165 9613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:33.053117 9613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--max_cell_size_bytes=1000000
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_async_preallocate_segments=false
--log_segment_size_mb=1
--raft_prepare_replacement_before_eviction=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:32897
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=46737
--tserver_master_addrs=127.1.109.126:36005
--never_fsync=true
--heap_profile_path=/tmp/kudu.9613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--log_target_replay_size_mb=1
--maintenance_manager_polling_interval_ms=100
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:33.054172 9613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:33.055366 9613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:33.063457 9619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:33.064163 9622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:33.064692 9620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:33.065662 9613 server_base.cc:1034] running on GCE node
I20250114 03:05:33.066015 9613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:33.066540 9613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:33.067719 9613 hybrid_clock.cc:648] HybridClock initialized: now 1736823933067705 us; error 27 us; skew 500 ppm
I20250114 03:05:33.069738 9613 webserver.cc:458] Webserver started at http://127.1.109.66:46737/ using document root <none> and password file <none>
I20250114 03:05:33.070322 9613 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:33.070398 9613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:33.074007 9613 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:33.076012 9628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:33.077060 9613 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:33.077165 9613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "82e12f0b53bc4e4fb39686d5d9684954"
format_stamp: "Formatted at 2025-01-14 03:05:21 on dist-test-slave-r16g"
I20250114 03:05:33.077581 9613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:33.103447 9613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:33.104140 9613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:33.104326 9613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:33.104969 9613 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:33.106093 9635 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20250114 03:05:33.109006 9635 ts_tablet_manager.cc:1852] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_COPYING
I20250114 03:05:33.110450 9635 ts_tablet_manager.cc:1905] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:33.113593 9635 ts_tablet_manager.cc:1918] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:33.114478 9613 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20250114 03:05:33.114557 9613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.000s
I20250114 03:05:33.114634 9613 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:33.114665 9613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:33.123843 9613 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:32897
I20250114 03:05:33.123871 9742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:32897 every 8 connection(s)
I20250114 03:05:33.124856 9613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:33.133023 9743 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36005
I20250114 03:05:33.133217 9743 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:33.133615 9743 heartbeater.cc:510] Master 127.1.109.126:36005 requested a full tablet report, sending...
I20250114 03:05:33.134779 9417 ts_manager.cc:194] Registered new tserver with Master: 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:33.135015 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9613
I20250114 03:05:33.135159 1461 test_util.cc:274] Using random seed: -711825098
I20250114 03:05:33.135530 9417 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:51207
I20250114 03:05:33.145239 1461 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
W20250114 03:05:33.170187 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.170187 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.170225 9522 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.170225 9520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.176712 9747 meta_cache.cc:302] tablet a8c2353cd1d84f0c92405d73bd142a1a: replica 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359) has failed: Network error: Client connection negotiation failed: client connection to 127.1.109.65:38359: connect: Connection refused (error 111)
W20250114 03:05:33.188436 9520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.197559 9520 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.197628 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.197829 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.205165 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.218245 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.223881 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.234366 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.245949 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.261086 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.275180 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.286727 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.301586 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.309352 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.327243 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.355854 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.356482 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.375221 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.382376 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.418124 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.424268 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.441815 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.462724 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.491083 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.491076 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.516315 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.516299 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.542726 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.552076 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.577909 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.604905 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.630475 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.643095 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.671130 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.696653 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.730084 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.730347 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.763283 9519 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.763288 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.801653 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.840858 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.868748 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.887574 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.921240 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.958670 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:33.985543 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.027699 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.049257 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.082270 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.122566 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
I20250114 03:05:34.137799 9743 heartbeater.cc:502] Master 127.1.109.126:36005 was elected leader, sending a full tablet report...
W20250114 03:05:34.155925 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.183385 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.200963 9747 meta_cache.cc:302] tablet a8c2353cd1d84f0c92405d73bd142a1a: replica 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359) has failed: Network error: Client connection negotiation failed: client connection to 127.1.109.65:38359: connect: Connection refused (error 111)
W20250114 03:05:34.221145 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.254712 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.294744 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.323302 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.368150 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.399816 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.445600 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.475906 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
W20250114 03:05:34.530278 9521 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:38920: Illegal state: replica e6648bf13d084e3a9c177fd7d38bc6e6 is not leader of this config: current role FOLLOWER
I20250114 03:05:34.532148 9758 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:05:34.532315 9758 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:34.533373 9758 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
W20250114 03:05:34.534055 9495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.65:38359: connect: Connection refused (error 111)
W20250114 03:05:34.534891 9495 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359): Network error: Client connection negotiation failed: client connection to 127.1.109.65:38359: connect: Connection refused (error 111)
I20250114 03:05:34.537564 9697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "82e12f0b53bc4e4fb39686d5d9684954" is_pre_election: true
I20250114 03:05:34.538398 9697 raft_consensus.cc:1798] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20250114 03:05:34.538719 9697 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6648bf13d084e3a9c177fd7d38bc6e6 in term 3.
I20250114 03:05:34.539115 9496 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e12f0b53bc4e4fb39686d5d9684954, e6648bf13d084e3a9c177fd7d38bc6e6; no voters: 62c797d1626f4638990c88ecc0aa2ad8
I20250114 03:05:34.539348 9758 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250114 03:05:34.539418 9758 raft_consensus.cc:491] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:05:34.539497 9758 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 3 FOLLOWER]: Advancing to term 4
I20250114 03:05:34.541118 9758 raft_consensus.cc:513] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:34.541469 9758 leader_election.cc:290] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 4 election: Requested vote from peers 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359), 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897)
I20250114 03:05:34.541831 9697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8c2353cd1d84f0c92405d73bd142a1a" candidate_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" candidate_term: 4 candidate_status { last_received { term: 3 index: 44 } } ignore_live_leader: false dest_uuid: "82e12f0b53bc4e4fb39686d5d9684954"
I20250114 03:05:34.541925 9697 raft_consensus.cc:1798] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: voting while tombstoned based on last-logged opid 1.1
I20250114 03:05:34.542130 9697 raft_consensus.cc:3054] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 3 FOLLOWER]: Advancing to term 4
W20250114 03:05:34.542389 9495 leader_election.cc:336] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359): Network error: Client connection negotiation failed: client connection to 127.1.109.65:38359: connect: Connection refused (error 111)
I20250114 03:05:34.543143 9697 raft_consensus.cc:2463] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6648bf13d084e3a9c177fd7d38bc6e6 in term 4.
I20250114 03:05:34.543444 9496 leader_election.cc:304] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 82e12f0b53bc4e4fb39686d5d9684954, e6648bf13d084e3a9c177fd7d38bc6e6; no voters: 62c797d1626f4638990c88ecc0aa2ad8
I20250114 03:05:34.543620 9758 raft_consensus.cc:2798] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 4 FOLLOWER]: Leader election won for term 4
I20250114 03:05:34.543834 9758 raft_consensus.cc:695] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [term 4 LEADER]: Becoming Leader. State: Replica: e6648bf13d084e3a9c177fd7d38bc6e6, State: Running, Role: LEADER
I20250114 03:05:34.544162 9758 consensus_queue.cc:237] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 3.44, Last appended by leader: 44, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:34.547137 9416 catalog_manager.cc:5526] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 reported cstate change: term changed from 3 to 4. New cstate: current_term: 4 leader_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" committed_config { opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } } }
W20250114 03:05:34.579849 9496 consensus_peers.cc:487] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 -> Peer 82e12f0b53bc4e4fb39686d5d9684954 (127.1.109.66:32897): Couldn't send request to peer 82e12f0b53bc4e4fb39686d5d9684954. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250114 03:05:34.579933 9495 consensus_peers.cc:487] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6 -> Peer 62c797d1626f4638990c88ecc0aa2ad8 (127.1.109.65:38359): Couldn't send request to peer 62c797d1626f4638990c88ecc0aa2ad8. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.65:38359: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250114 03:05:35.077769 9770 tablet_replica.cc:331] stopping tablet replica
I20250114 03:05:35.077967 9770 raft_consensus.cc:2238] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 4 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:35.078058 9770 raft_consensus.cc:2267] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:35.078222 9770 ts_tablet_manager.cc:927] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Initiating tablet copy from peer e6648bf13d084e3a9c177fd7d38bc6e6 (127.1.109.67:46041)
I20250114 03:05:35.078814 9770 tablet_copy_client.cc:323] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet copy: Beginning tablet copy session from remote peer at address 127.1.109.67:46041
I20250114 03:05:35.082998 9582 tablet_copy_service.cc:140] P e6648bf13d084e3a9c177fd7d38bc6e6: Received BeginTabletCopySession request for tablet a8c2353cd1d84f0c92405d73bd142a1a from peer 82e12f0b53bc4e4fb39686d5d9684954 ({username='slave'} at 127.1.109.66:51499)
I20250114 03:05:35.083150 9582 tablet_copy_service.cc:161] P e6648bf13d084e3a9c177fd7d38bc6e6: Beginning new tablet copy session on tablet a8c2353cd1d84f0c92405d73bd142a1a from peer 82e12f0b53bc4e4fb39686d5d9684954 at {username='slave'} at 127.1.109.66:51499: session id = 82e12f0b53bc4e4fb39686d5d9684954-a8c2353cd1d84f0c92405d73bd142a1a
I20250114 03:05:35.084544 9582 tablet_copy_source_session.cc:215] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Tablet Copy: opened 5 blocks and 3 log segments
I20250114 03:05:35.086583 9770 ts_tablet_manager.cc:1905] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Deleting tablet data with delete state TABLET_DATA_COPYING
I20250114 03:05:35.088800 9770 ts_tablet_manager.cc:1918] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.1
I20250114 03:05:35.089242 9770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8c2353cd1d84f0c92405d73bd142a1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:35.090929 9770 tablet_copy_client.cc:806] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet copy: Starting download of 5 data blocks...
I20250114 03:05:35.098651 9770 tablet_copy_client.cc:670] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet copy: Starting download of 3 WAL segments...
I20250114 03:05:35.108115 9770 tablet_copy_client.cc:538] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250114 03:05:35.110323 9770 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Bootstrap starting.
I20250114 03:05:35.118425 9770 log.cc:826] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:35.125945 9770 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Bootstrap replayed 1/3 log segments. Stats: ops{read=7 overwritten=0 applied=6 ignored=6} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250114 03:05:35.130628 9770 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Bootstrap replayed 2/3 log segments. Stats: ops{read=14 overwritten=0 applied=13 ignored=13} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20250114 03:05:35.135597 9770 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Bootstrap replayed 3/3 log segments. Stats: ops{read=26 overwritten=0 applied=21 ignored=19} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 5 replicates
I20250114 03:05:35.136132 9770 tablet_bootstrap.cc:492] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Bootstrap complete.
I20250114 03:05:35.137025 9770 ts_tablet_manager.cc:1397] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.003s
I20250114 03:05:35.138608 9770 raft_consensus.cc:357] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 4 FOLLOWER]: Replica starting. Triggering 5 pending ops. Active config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:35.140610 9770 raft_consensus.cc:738] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 4 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e12f0b53bc4e4fb39686d5d9684954, State: Initialized, Role: FOLLOWER
I20250114 03:05:35.141160 9770 consensus_queue.cc:260] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 44, Last appended: 4.49, Last appended by leader: 49, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 44 OBSOLETE_local: false peers { permanent_uuid: "62c797d1626f4638990c88ecc0aa2ad8" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38359 } } peers { permanent_uuid: "e6648bf13d084e3a9c177fd7d38bc6e6" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 46041 } } peers { permanent_uuid: "82e12f0b53bc4e4fb39686d5d9684954" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 32897 } attrs { } }
I20250114 03:05:35.142081 9770 ts_tablet_manager.cc:1428] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20250114 03:05:35.142825 9582 tablet_copy_service.cc:342] P e6648bf13d084e3a9c177fd7d38bc6e6: Request end of tablet copy session 82e12f0b53bc4e4fb39686d5d9684954-a8c2353cd1d84f0c92405d73bd142a1a received from {username='slave'} at 127.1.109.66:51499
I20250114 03:05:35.142962 9582 tablet_copy_service.cc:434] P e6648bf13d084e3a9c177fd7d38bc6e6: ending tablet copy session 82e12f0b53bc4e4fb39686d5d9684954-a8c2353cd1d84f0c92405d73bd142a1a on tablet a8c2353cd1d84f0c92405d73bd142a1a with peer 82e12f0b53bc4e4fb39686d5d9684954
I20250114 03:05:35.230463 9744 maintenance_manager.cc:418] P 82e12f0b53bc4e4fb39686d5d9684954: Scheduling UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a): 579 bytes on disk
I20250114 03:05:35.231521 9633 maintenance_manager.cc:642] P 82e12f0b53bc4e4fb39686d5d9684954: UndoDeltaBlockGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20250114 03:05:35.581597 9697 raft_consensus.cc:1212] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954 [term 4 FOLLOWER]: Deduplicated request from leader. Original: 3.44->[4.45-4.49] Dedup: 4.49->[]
I20250114 03:05:35.584252 9777 mvcc.cc:204] Tried to move back new op lower bound from 7114030836129304576 to 7114030835895709696. Current Snapshot: MvccSnapshot[applied={T|T < 7114030836031590400}]
I20250114 03:05:35.590553 9783 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:38920 (ReqId={client: 0a7b6d43d838465cb742a31301c76904, seq_no=0, attempt_no=49}) took 1014 ms. Trace:
I20250114 03:05:35.590677 9783 rpcz_store.cc:276] 0114 03:05:34.575616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:34.575720 (+ 104us) service_pool.cc:225] Handling call
0114 03:05:35.590534 (+1014814us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:34.576429 (+ 0us) write_op.cc:183] PREPARE: starting on tablet a8c2353cd1d84f0c92405d73bd142a1a
0114 03:05:34.576587 (+ 158us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:34.576596 (+ 9us) write_op.cc:435] Acquired schema lock
0114 03:05:34.576597 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:34.577281 (+ 684us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:34.577330 (+ 49us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:34.577335 (+ 5us) tablet.cc:682] Acquiring locks for 50 operations
0114 03:05:34.578004 (+ 669us) tablet.cc:698] Row locks acquired
0114 03:05:34.578005 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:34.578063 (+ 58us) write_op.cc:270] Start()
0114 03:05:34.578098 (+ 35us) write_op.cc:276] Timestamp: P: 1736823934578025 usec, L: 0
0114 03:05:34.578101 (+ 3us) op_driver.cc:348] REPLICATION: starting
0114 03:05:34.578274 (+ 173us) log.cc:844] Serialized 2455 byte log entry
0114 03:05:35.583517 (+1005243us) op_driver.cc:464] REPLICATION: finished
0114 03:05:35.583751 (+ 234us) write_op.cc:301] APPLY: starting
0114 03:05:35.583775 (+ 24us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:35.585213 (+ 1438us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:35.585239 (+ 26us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:35.589968 (+ 4729us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:35.590024 (+ 56us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=46,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:35.590030 (+ 6us) write_op.cc:312] APPLY: finished
0114 03:05:35.590147 (+ 117us) log.cc:844] Serialized 319 byte log entry
0114 03:05:35.590417 (+ 270us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:35.590451 (+ 34us) write_op.cc:454] Released schema lock
0114 03:05:35.590459 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":154,"cfile_cache_miss":2,"cfile_cache_miss_bytes":4127,"cfile_init":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":6,"mutex_wait_us":246,"num_ops":50,"prepare.queue_time_us":262,"prepare.run_cpu_time_us":2008,"prepare.run_wall_time_us":2009,"raft.queue_time_us":96,"raft.run_cpu_time_us":793,"raft.run_wall_time_us":810,"replication_time_us":1005355,"spinlock_wait_cycles":519680,"thread_start_us":198,"threads_started":2}]]}
I20250114 03:05:36.136961 9744 maintenance_manager.cc:418] P 82e12f0b53bc4e4fb39686d5d9684954: Scheduling LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a): free 920430 bytes of WAL
I20250114 03:05:36.137310 9633 log_reader.cc:385] T a8c2353cd1d84f0c92405d73bd142a1a: removed 1 log segments from log reader
I20250114 03:05:36.137387 9633 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P 82e12f0b53bc4e4fb39686d5d9684954: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000001 (ops 24-30)
I20250114 03:05:36.137910 9633 maintenance_manager.cc:642] P 82e12f0b53bc4e4fb39686d5d9684954: LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20250114 03:05:36.139842 9609 maintenance_manager.cc:418] P e6648bf13d084e3a9c177fd7d38bc6e6: Scheduling LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a): free 920430 bytes of WAL
I20250114 03:05:36.140211 9498 log_reader.cc:385] T a8c2353cd1d84f0c92405d73bd142a1a: removed 1 log segments from log reader
I20250114 03:05:36.140295 9498 log.cc:1078] T a8c2353cd1d84f0c92405d73bd142a1a P e6648bf13d084e3a9c177fd7d38bc6e6: Deleting log segment in path: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/a8c2353cd1d84f0c92405d73bd142a1a/wal-000000001 (ops 24-30)
I20250114 03:05:36.140780 9498 maintenance_manager.cc:642] P e6648bf13d084e3a9c177fd7d38bc6e6: LogGCOp(a8c2353cd1d84f0c92405d73bd142a1a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":47360}
I20250114 03:05:36.178524 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 9613
I20250114 03:05:36.188649 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 9455
I20250114 03:05:36.196059 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 9386
2025-01-14T03:05:36Z chronyd exiting
[ OK ] RaftConsensusElectionITest.TombstonedVoteAfterFailedTabletCopy (15514 ms)
[ RUN ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL
I20250114 03:05:36.221226 1461 test_util.cc:274] Using random seed: -708739036
I20250114 03:05:36.222034 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:36.222079 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:36.222090 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:05:36.222100 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:05:36.222111 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:36Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:36Z Disabled control of system clock
I20250114 03:05:36.230933 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:36755
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:36755 with env {}
W20250114 03:05:36.351274 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:36.351579 9805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:36.351663 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:36.355145 9805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:36.355226 9805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:36.355263 9805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:36.355296 9805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:36.359386 9805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:36755
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:36755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.9805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:36.360667 9805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:36.361960 9805 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:36.370048 9813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:36.370692 9810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:36.371033 9811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:36.372041 9805 server_base.cc:1034] running on GCE node
I20250114 03:05:36.372658 9805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:36.373756 9805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:36.374924 9805 hybrid_clock.cc:648] HybridClock initialized: now 1736823936374900 us; error 37 us; skew 500 ppm
I20250114 03:05:36.381964 9805 webserver.cc:458] Webserver started at http://127.1.109.126:42009/ using document root <none> and password file <none>
I20250114 03:05:36.382647 9805 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:36.382725 9805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:36.382907 9805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:36.384786 9805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e420c071deca47b3b5d2daaf42107a6e"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.385349 9805 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e420c071deca47b3b5d2daaf42107a6e"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.389109 9805 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20250114 03:05:36.391640 9819 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.392774 9805 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:36.392937 9805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e420c071deca47b3b5d2daaf42107a6e"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.393041 9805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:36.411410 9805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:36.412146 9805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:36.412322 9805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:36.419308 9805 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:36755
I20250114 03:05:36.419320 9871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:36755 every 8 connection(s)
I20250114 03:05:36.420383 9805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:36.423218 9872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:36.427011 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9805
I20250114 03:05:36.427130 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:36.429209 9872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e: Bootstrap starting.
I20250114 03:05:36.432137 9872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:36.433115 9872 log.cc:826] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:36.435302 9872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e: No bootstrap required, opened a new log
I20250114 03:05:36.439111 9872 raft_consensus.cc:357] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e420c071deca47b3b5d2daaf42107a6e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36755 } }
I20250114 03:05:36.439318 9872 raft_consensus.cc:383] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:36.439354 9872 raft_consensus.cc:738] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e420c071deca47b3b5d2daaf42107a6e, State: Initialized, Role: FOLLOWER
I20250114 03:05:36.439792 9872 consensus_queue.cc:260] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e420c071deca47b3b5d2daaf42107a6e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36755 } }
I20250114 03:05:36.439934 9872 raft_consensus.cc:397] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:36.439980 9872 raft_consensus.cc:491] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:36.440068 9872 raft_consensus.cc:3054] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:36.440924 9872 raft_consensus.cc:513] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e420c071deca47b3b5d2daaf42107a6e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36755 } }
I20250114 03:05:36.441296 9872 leader_election.cc:304] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e420c071deca47b3b5d2daaf42107a6e; no voters:
I20250114 03:05:36.441632 9872 leader_election.cc:290] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:36.441718 9877 raft_consensus.cc:2798] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:36.441962 9877 raft_consensus.cc:695] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [term 1 LEADER]: Becoming Leader. State: Replica: e420c071deca47b3b5d2daaf42107a6e, State: Running, Role: LEADER
I20250114 03:05:36.442293 9877 consensus_queue.cc:237] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e420c071deca47b3b5d2daaf42107a6e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36755 } }
I20250114 03:05:36.442790 9872 sys_catalog.cc:564] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:36.444320 9879 sys_catalog.cc:455] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e420c071deca47b3b5d2daaf42107a6e. Latest consensus state: current_term: 1 leader_uuid: "e420c071deca47b3b5d2daaf42107a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e420c071deca47b3b5d2daaf42107a6e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36755 } } }
I20250114 03:05:36.444465 9879 sys_catalog.cc:458] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:36.444800 9878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e420c071deca47b3b5d2daaf42107a6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e420c071deca47b3b5d2daaf42107a6e" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 36755 } } }
I20250114 03:05:36.444931 9878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:36.445245 9886 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:36.447804 9886 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:36.453469 9886 catalog_manager.cc:1348] Generated new cluster ID: 3339fd2aafd345ed82fb2dd02c2670c1
I20250114 03:05:36.453544 9886 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:36.472160 9886 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:36.473320 9886 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:36.481920 9886 catalog_manager.cc:5899] T 00000000000000000000000000000000 P e420c071deca47b3b5d2daaf42107a6e: Generated new TSK 0
I20250114 03:05:36.482699 9886 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:36.494977 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36755
--builtin_ntp_servers=127.1.109.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
W20250114 03:05:36.616149 9896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:36.616410 9896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:36.616458 9896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:36.619871 9896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:36.619999 9896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:36.624241 9896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36755
--never_fsync=true
--heap_profile_path=/tmp/kudu.9896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:36.625432 9896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:36.626612 9896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:36.635072 9904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:36.635736 9901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:36.636348 9902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:36.637231 9896 server_base.cc:1034] running on GCE node
I20250114 03:05:36.637647 9896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:36.638335 9896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:36.639510 9896 hybrid_clock.cc:648] HybridClock initialized: now 1736823936639494 us; error 29 us; skew 500 ppm
I20250114 03:05:36.644237 9896 webserver.cc:458] Webserver started at http://127.1.109.65:33281/ using document root <none> and password file <none>
I20250114 03:05:36.645049 9896 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:36.645112 9896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:36.645295 9896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:36.647119 9896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3a9a4ca552714a1abc8aba425b3420a9"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.647667 9896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3a9a4ca552714a1abc8aba425b3420a9"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.651573 9896 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20250114 03:05:36.654060 9910 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.655331 9896 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:36.655496 9896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3a9a4ca552714a1abc8aba425b3420a9"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.655601 9896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:36.679525 9896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:36.680299 9896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:36.680505 9896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:36.681231 9896 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:36.682308 9896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:36.682366 9896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.682440 9896 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:36.682471 9896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.696095 9896 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:38863
I20250114 03:05:36.696131 10023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:38863 every 8 connection(s)
I20250114 03:05:36.697288 9896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:36.701850 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 9896
I20250114 03:05:36.701989 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:36.704596 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36755
--builtin_ntp_servers=127.1.109.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20250114 03:05:36.709017 10024 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36755
I20250114 03:05:36.709335 10024 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:36.710104 10024 heartbeater.cc:510] Master 127.1.109.126:36755 requested a full tablet report, sending...
I20250114 03:05:36.711802 9836 ts_manager.cc:194] Registered new tserver with Master: 3a9a4ca552714a1abc8aba425b3420a9 (127.1.109.65:38863)
I20250114 03:05:36.713081 9836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:45419
W20250114 03:05:36.823478 10027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:36.823783 10027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:36.823864 10027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:36.827284 10027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:36.827462 10027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:36.831614 10027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36755
--never_fsync=true
--heap_profile_path=/tmp/kudu.10027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:36.832846 10027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:36.834098 10027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:36.843092 10032 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:36.843362 10035 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:36.844071 10033 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:36.845088 10027 server_base.cc:1034] running on GCE node
I20250114 03:05:36.845604 10027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:36.846326 10027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:36.847506 10027 hybrid_clock.cc:648] HybridClock initialized: now 1736823936847486 us; error 37 us; skew 500 ppm
I20250114 03:05:36.849786 10027 webserver.cc:458] Webserver started at http://127.1.109.66:35391/ using document root <none> and password file <none>
I20250114 03:05:36.850510 10027 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:36.850577 10027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:36.850782 10027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:36.852675 10027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b2b948e36095485faa6060007121083a"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.853260 10027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b2b948e36095485faa6060007121083a"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.857270 10027 fs_manager.cc:696] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20250114 03:05:36.859704 10041 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.860918 10027 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:36.861060 10027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b2b948e36095485faa6060007121083a"
format_stamp: "Formatted at 2025-01-14 03:05:36 on dist-test-slave-r16g"
I20250114 03:05:36.861181 10027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:36.877444 10027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:36.878202 10027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:36.878382 10027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:36.879004 10027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:36.879962 10027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:36.880012 10027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.880079 10027 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:36.880117 10027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:36.889262 10027 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:39785
I20250114 03:05:36.889282 10154 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:39785 every 8 connection(s)
I20250114 03:05:36.890328 10027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:36.892009 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10027
I20250114 03:05:36.892117 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:36.895118 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36755
--builtin_ntp_servers=127.1.109.84:43751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--leader_failure_max_missed_heartbeat_periods=3 with env {}
I20250114 03:05:36.904664 10155 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36755
I20250114 03:05:36.904958 10155 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:36.905567 10155 heartbeater.cc:510] Master 127.1.109.126:36755 requested a full tablet report, sending...
I20250114 03:05:36.906741 9836 ts_manager.cc:194] Registered new tserver with Master: b2b948e36095485faa6060007121083a (127.1.109.66:39785)
I20250114 03:05:36.907644 9836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:47575
W20250114 03:05:37.017793 10158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:37.018056 10158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:37.018103 10158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:37.021293 10158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:37.021463 10158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:37.025417 10158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:43751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_heartbeat_interval_ms=200
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:36755
--never_fsync=true
--heap_profile_path=/tmp/kudu.10158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:37.026561 10158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:37.027760 10158 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:37.036535 10163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:37.036705 10164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:37.037478 10166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:37.038273 10158 server_base.cc:1034] running on GCE node
I20250114 03:05:37.038743 10158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:37.039346 10158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:37.040518 10158 hybrid_clock.cc:648] HybridClock initialized: now 1736823937040487 us; error 44 us; skew 500 ppm
I20250114 03:05:37.042613 10158 webserver.cc:458] Webserver started at http://127.1.109.67:40135/ using document root <none> and password file <none>
I20250114 03:05:37.043190 10158 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:37.043246 10158 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:37.043423 10158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:37.045228 10158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bb2e127fa46647c6b868bc92d3fbc9e8"
format_stamp: "Formatted at 2025-01-14 03:05:37 on dist-test-slave-r16g"
I20250114 03:05:37.045720 10158 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bb2e127fa46647c6b868bc92d3fbc9e8"
format_stamp: "Formatted at 2025-01-14 03:05:37 on dist-test-slave-r16g"
I20250114 03:05:37.049247 10158 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20250114 03:05:37.051502 10172 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:37.052529 10158 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:37.052641 10158 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bb2e127fa46647c6b868bc92d3fbc9e8"
format_stamp: "Formatted at 2025-01-14 03:05:37 on dist-test-slave-r16g"
I20250114 03:05:37.052732 10158 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:37.073514 10158 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:37.074213 10158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:37.074398 10158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:37.074976 10158 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:37.075947 10158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:37.075995 10158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:37.076056 10158 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:37.076082 10158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:37.085654 10158 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:34583
I20250114 03:05:37.085692 10285 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:34583 every 8 connection(s)
I20250114 03:05:37.086700 10158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:37.095053 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10158
I20250114 03:05:37.095163 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.RaftConsensusElectionITest.DisruptiveServerAndSlowWAL.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:37.095636 10286 heartbeater.cc:346] Connected to a master server at 127.1.109.126:36755
I20250114 03:05:37.095883 10286 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:37.096594 10286 heartbeater.cc:510] Master 127.1.109.126:36755 requested a full tablet report, sending...
I20250114 03:05:37.097770 9836 ts_manager.cc:194] Registered new tserver with Master: bb2e127fa46647c6b868bc92d3fbc9e8 (127.1.109.67:34583)
I20250114 03:05:37.098317 9836 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:40997
I20250114 03:05:37.109539 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:37.118769 9836 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51968:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250114 03:05:37.121482 9836 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:05:37.138128 10220 tablet_service.cc:1467] Processing CreateTablet for tablet e9d59cec93c04246b9b519e39015aa59 (DEFAULT_TABLE table=TestTable [id=a144ec396b794f178cc373b06da33cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:37.138128 9958 tablet_service.cc:1467] Processing CreateTablet for tablet e9d59cec93c04246b9b519e39015aa59 (DEFAULT_TABLE table=TestTable [id=a144ec396b794f178cc373b06da33cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:37.139247 10220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d59cec93c04246b9b519e39015aa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:37.139247 9958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d59cec93c04246b9b519e39015aa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:37.139273 10089 tablet_service.cc:1467] Processing CreateTablet for tablet e9d59cec93c04246b9b519e39015aa59 (DEFAULT_TABLE table=TestTable [id=a144ec396b794f178cc373b06da33cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250114 03:05:37.140120 10089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9d59cec93c04246b9b519e39015aa59. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:37.146000 10305 tablet_bootstrap.cc:492] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Bootstrap starting.
I20250114 03:05:37.146664 10304 tablet_bootstrap.cc:492] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Bootstrap starting.
I20250114 03:05:37.146663 10303 tablet_bootstrap.cc:492] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a: Bootstrap starting.
I20250114 03:05:37.148631 10305 tablet_bootstrap.cc:654] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:37.148806 10303 tablet_bootstrap.cc:654] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:37.148806 10304 tablet_bootstrap.cc:654] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:37.149447 10305 log.cc:826] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:37.149641 10304 log.cc:826] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:37.149641 10303 log.cc:826] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:37.151417 10303 tablet_bootstrap.cc:492] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a: No bootstrap required, opened a new log
I20250114 03:05:37.151437 10304 tablet_bootstrap.cc:492] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: No bootstrap required, opened a new log
I20250114 03:05:37.151437 10305 tablet_bootstrap.cc:492] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: No bootstrap required, opened a new log
I20250114 03:05:37.151609 10303 ts_tablet_manager.cc:1397] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250114 03:05:37.151610 10304 ts_tablet_manager.cc:1397] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20250114 03:05:37.151610 10305 ts_tablet_manager.cc:1397] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20250114 03:05:37.155005 10303 raft_consensus.cc:357] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.155215 10303 raft_consensus.cc:383] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:37.155277 10303 raft_consensus.cc:738] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2b948e36095485faa6060007121083a, State: Initialized, Role: FOLLOWER
I20250114 03:05:37.155839 10303 consensus_queue.cc:260] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.155843 10304 raft_consensus.cc:357] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.155843 10305 raft_consensus.cc:357] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.156060 10305 raft_consensus.cc:383] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:37.156060 10304 raft_consensus.cc:383] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:37.156109 10305 raft_consensus.cc:738] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2e127fa46647c6b868bc92d3fbc9e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:37.156117 10304 raft_consensus.cc:738] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a9a4ca552714a1abc8aba425b3420a9, State: Initialized, Role: FOLLOWER
I20250114 03:05:37.156553 10304 consensus_queue.cc:260] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.156553 10305 consensus_queue.cc:260] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.156776 10303 ts_tablet_manager.cc:1428] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20250114 03:05:37.157212 10155 heartbeater.cc:502] Master 127.1.109.126:36755 was elected leader, sending a full tablet report...
I20250114 03:05:37.157460 10304 ts_tablet_manager.cc:1428] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20250114 03:05:37.157438 10305 ts_tablet_manager.cc:1428] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250114 03:05:37.157703 10286 heartbeater.cc:502] Master 127.1.109.126:36755 was elected leader, sending a full tablet report...
I20250114 03:05:37.158416 10024 heartbeater.cc:502] Master 127.1.109.126:36755 was elected leader, sending a full tablet report...
I20250114 03:05:37.194882 10311 raft_consensus.cc:491] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250114 03:05:37.195077 10311 raft_consensus.cc:513] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.196172 10311 leader_election.cc:290] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2b948e36095485faa6060007121083a (127.1.109.66:39785), 3a9a4ca552714a1abc8aba425b3420a9 (127.1.109.65:38863)
W20250114 03:05:37.198495 10025 tablet.cc:2367] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:37.200120 10109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b948e36095485faa6060007121083a" is_pre_election: true
I20250114 03:05:37.200273 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: true
I20250114 03:05:37.200428 10109 raft_consensus.cc:2463] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb2e127fa46647c6b868bc92d3fbc9e8 in term 0.
I20250114 03:05:37.200567 9978 raft_consensus.cc:2463] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bb2e127fa46647c6b868bc92d3fbc9e8 in term 0.
I20250114 03:05:37.200997 10173 leader_election.cc:304] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2b948e36095485faa6060007121083a, bb2e127fa46647c6b868bc92d3fbc9e8; no voters:
I20250114 03:05:37.201263 10311 raft_consensus.cc:2798] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250114 03:05:37.201339 10311 raft_consensus.cc:491] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250114 03:05:37.201380 10311 raft_consensus.cc:3054] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:37.202301 10311 raft_consensus.cc:513] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.202662 10311 leader_election.cc:290] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [CANDIDATE]: Term 1 election: Requested vote from peers b2b948e36095485faa6060007121083a (127.1.109.66:39785), 3a9a4ca552714a1abc8aba425b3420a9 (127.1.109.65:38863)
I20250114 03:05:37.203033 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9"
I20250114 03:05:37.203049 10109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2b948e36095485faa6060007121083a"
I20250114 03:05:37.203162 9978 raft_consensus.cc:3054] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:37.203168 10109 raft_consensus.cc:3054] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:37.204070 9978 raft_consensus.cc:2463] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb2e127fa46647c6b868bc92d3fbc9e8 in term 1.
I20250114 03:05:37.204097 10109 raft_consensus.cc:2463] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bb2e127fa46647c6b868bc92d3fbc9e8 in term 1.
I20250114 03:05:37.204372 10174 leader_election.cc:304] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a9a4ca552714a1abc8aba425b3420a9, bb2e127fa46647c6b868bc92d3fbc9e8; no voters:
I20250114 03:05:37.204551 10311 raft_consensus.cc:2798] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:37.204748 10311 raft_consensus.cc:695] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [term 1 LEADER]: Becoming Leader. State: Replica: bb2e127fa46647c6b868bc92d3fbc9e8, State: Running, Role: LEADER
I20250114 03:05:37.205101 10311 consensus_queue.cc:237] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } }
I20250114 03:05:37.208566 9836 catalog_manager.cc:5526] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to bb2e127fa46647c6b868bc92d3fbc9e8 (127.1.109.67). New cstate: current_term: 1 leader_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 34583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 } health_report { overall_health: UNKNOWN } } }
I20250114 03:05:37.231595 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:37.233065 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3a9a4ca552714a1abc8aba425b3420a9 to finish bootstrapping
I20250114 03:05:37.238468 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b2b948e36095485faa6060007121083a to finish bootstrapping
I20250114 03:05:37.243726 1461 ts_itest-base.cc:233] Waiting for 1 tablets on tserver bb2e127fa46647c6b868bc92d3fbc9e8 to finish bootstrapping
I20250114 03:05:37.249516 10018 generic_service.cc:175] {username='slave'} at 127.0.0.1:50192 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20250114 03:05:37.250504 10149 generic_service.cc:175] {username='slave'} at 127.0.0.1:36802 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20250114 03:05:37.251492 10280 generic_service.cc:175] {username='slave'} at 127.0.0.1:58548 changed flags via RPC: log_inject_latency from 'false' to 'false'
I20250114 03:05:37.262749 9978 raft_consensus.cc:1270] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9 [term 1 FOLLOWER]: Refusing update from remote peer bb2e127fa46647c6b868bc92d3fbc9e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:37.262749 10109 raft_consensus.cc:1270] T e9d59cec93c04246b9b519e39015aa59 P b2b948e36095485faa6060007121083a [term 1 FOLLOWER]: Refusing update from remote peer bb2e127fa46647c6b868bc92d3fbc9e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250114 03:05:37.263381 10311 consensus_queue.cc:1035] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a9a4ca552714a1abc8aba425b3420a9" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 38863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:37.263557 10313 consensus_queue.cc:1035] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2b948e36095485faa6060007121083a" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 39785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250114 03:05:37.269414 10317 mvcc.cc:204] Tried to move back new op lower bound from 7114030847021842432 to 7114030846795001856. Current Snapshot: MvccSnapshot[applied={T|T < 7114030847021842432}]
I20250114 03:05:37.270126 10319 mvcc.cc:204] Tried to move back new op lower bound from 7114030847021842432 to 7114030846795001856. Current Snapshot: MvccSnapshot[applied={T|T < 7114030847021842432}]
I20250114 03:05:37.283318 10280 generic_service.cc:175] {username='slave'} at 127.0.0.1:58548 changed flags via RPC: log_inject_latency_ms_mean from '100' to '1200'
I20250114 03:05:37.284128 10280 generic_service.cc:175] {username='slave'} at 127.0.0.1:58548 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20250114 03:05:37.284919 10280 generic_service.cc:175] {username='slave'} at 127.0.0.1:58548 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20250114 03:05:37.285674 10018 generic_service.cc:175] {username='slave'} at 127.0.0.1:50192 changed flags via RPC: log_inject_latency_ms_mean from '100' to '600'
I20250114 03:05:37.286317 10018 generic_service.cc:175] {username='slave'} at 127.0.0.1:50192 changed flags via RPC: log_inject_latency_ms_stddev from '100' to '0'
I20250114 03:05:37.286952 10018 generic_service.cc:175] {username='slave'} at 127.0.0.1:50192 changed flags via RPC: log_inject_latency from 'false' to 'true'
I20250114 03:05:37.287222 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10027
I20250114 03:05:37.298065 1461 test_util.cc:274] Using random seed: -707662193
I20250114 03:05:37.306762 1461 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
W20250114 03:05:37.314654 10314 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250114 03:05:37.315268 10173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111)
W20250114 03:05:37.316120 10173 consensus_peers.cc:487] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 -> Peer b2b948e36095485faa6060007121083a (127.1.109.66:39785): Couldn't send request to peer b2b948e36095485faa6060007121083a. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250114 03:05:37.316566 10321 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250114 03:05:37.357928 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.359663 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 2 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.411175 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.412546 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 3 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.464071 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.465392 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 4 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.516956 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.518621 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 5 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.570279 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.571627 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 6 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.623083 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.624308 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 7 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.676025 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.677347 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 8 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.729053 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.730346 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 9 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.781926 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.783215 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 10 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.834829 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.836102 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 11 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.887761 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.889065 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 12 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.940735 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.942127 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 13 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:37.993736 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:37.995039 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 14 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.046541 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.047811 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 15 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.099465 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.100855 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 16 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.152478 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.153750 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 17 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.205358 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.206676 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 18 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.258211 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.259511 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 19 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.311110 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.312335 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 20 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:38.332051 10173 consensus_peers.cc:487] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 -> Peer b2b948e36095485faa6060007121083a (127.1.109.66:39785): Couldn't send request to peer b2b948e36095485faa6060007121083a. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250114 03:05:38.363911 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.365214 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 21 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.416680 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.417927 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 22 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.469493 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.470733 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 23 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.516052 10334 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58566 (ReqId={client: 01b740fea92940d0a85b2221e055b441, seq_no=0, attempt_no=0}) took 1203 ms. Trace:
I20250114 03:05:38.516151 10334 rpcz_store.cc:276] 0114 03:05:37.312727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:37.312785 (+ 58us) service_pool.cc:225] Handling call
0114 03:05:38.516027 (+1203242us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:37.313762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e9d59cec93c04246b9b519e39015aa59
0114 03:05:37.313848 (+ 86us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:37.313851 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:05:37.313852 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:37.313915 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:37.313922 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:37.313924 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:37.313956 (+ 32us) tablet.cc:698] Row locks acquired
0114 03:05:37.313957 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:37.314058 (+ 101us) write_op.cc:270] Start()
0114 03:05:37.314083 (+ 25us) write_op.cc:276] Timestamp: P: 1736823937314003 usec, L: 0
0114 03:05:37.314085 (+ 2us) op_driver.cc:348] REPLICATION: starting
0114 03:05:37.314290 (+ 205us) log.cc:844] Serialized 345 byte log entry
0114 03:05:38.515138 (+1200848us) op_driver.cc:464] REPLICATION: finished
0114 03:05:38.515366 (+ 228us) write_op.cc:301] APPLY: starting
0114 03:05:38.515385 (+ 19us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:38.515413 (+ 28us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:38.515428 (+ 15us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:38.515544 (+ 116us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:38.515569 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:38.515573 (+ 4us) write_op.cc:312] APPLY: finished
0114 03:05:38.515639 (+ 66us) log.cc:844] Serialized 22 byte log entry
0114 03:05:38.515726 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:38.515760 (+ 34us) write_op.cc:454] Released schema lock
0114 03:05:38.515768 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":164,"num_ops":1,"prepare.queue_time_us":46,"prepare.run_cpu_time_us":656,"prepare.run_wall_time_us":655,"raft.queue_time_us":122,"raft.run_cpu_time_us":995,"raft.run_wall_time_us":996,"replication_time_us":1201015,"spinlock_wait_cycles":82560,"thread_start_us":83,"threads_started":1}]]}
I20250114 03:05:38.522295 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.523857 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 24 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.575420 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.576673 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 25 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.628154 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.629356 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 26 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.680924 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.682129 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 27 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:38.719784 10314 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250114 03:05:38.721127 10321 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250114 03:05:38.733572 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.734783 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 28 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.786320 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.787536 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 29 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.838981 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.840145 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 30 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.891660 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.892886 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 31 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.944378 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.945622 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 32 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:38.997308 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:38.998577 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 33 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.050040 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.051239 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 34 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.102666 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.103859 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 35 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.155542 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.156803 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 36 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.208544 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.209793 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 37 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.261233 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.262389 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 38 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:39.276860 10173 consensus_peers.cc:487] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 -> Peer b2b948e36095485faa6060007121083a (127.1.109.66:39785): Couldn't send request to peer b2b948e36095485faa6060007121083a. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250114 03:05:39.313843 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.315021 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 39 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.366513 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.367717 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 40 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.419229 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.420583 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 41 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.472183 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.473423 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 42 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.524907 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.526273 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 43 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.577920 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.579134 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 44 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.630585 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.631834 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 45 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.683383 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.684617 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 46 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.736138 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.737412 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 47 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.788995 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.790359 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 48 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.841878 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.843113 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 49 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.894762 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.896021 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 50 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:39.921085 10340 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58566 (ReqId={client: 01b740fea92940d0a85b2221e055b441, seq_no=1, attempt_no=0}) took 1202 ms. Trace:
I20250114 03:05:39.921201 10340 rpcz_store.cc:276] 0114 03:05:38.718305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:38.718389 (+ 84us) service_pool.cc:225] Handling call
0114 03:05:39.921054 (+1202665us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:38.719137 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e9d59cec93c04246b9b519e39015aa59
0114 03:05:38.719227 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:38.719230 (+ 3us) write_op.cc:435] Acquired schema lock
0114 03:05:38.719232 (+ 2us) tablet.cc:659] Decoding operations
0114 03:05:38.719293 (+ 61us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:38.719302 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:38.719304 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:38.719336 (+ 32us) tablet.cc:698] Row locks acquired
0114 03:05:38.719337 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:38.719368 (+ 31us) write_op.cc:270] Start()
0114 03:05:38.719391 (+ 23us) write_op.cc:276] Timestamp: P: 1736823938719334 usec, L: 0
0114 03:05:38.719393 (+ 2us) op_driver.cc:348] REPLICATION: starting
0114 03:05:38.719549 (+ 156us) log.cc:844] Serialized 354 byte log entry
0114 03:05:39.920381 (+1200832us) op_driver.cc:464] REPLICATION: finished
0114 03:05:39.920624 (+ 243us) write_op.cc:301] APPLY: starting
0114 03:05:39.920642 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:39.920668 (+ 26us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:39.920686 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:39.920740 (+ 54us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:39.920764 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:39.920767 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:05:39.920864 (+ 97us) log.cc:844] Serialized 22 byte log entry
0114 03:05:39.920945 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:39.920961 (+ 16us) write_op.cc:454] Released schema lock
0114 03:05:39.920968 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"num_ops":1,"prepare.queue_time_us":200,"prepare.run_cpu_time_us":519,"prepare.run_wall_time_us":519,"raft.queue_time_us":88,"raft.run_cpu_time_us":239,"raft.run_wall_time_us":240,"replication_time_us":1200944,"thread_start_us":197,"threads_started":2}]]}
I20250114 03:05:39.947575 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:39.948729 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 51 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.000105 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.001333 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 52 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.052794 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.054088 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 53 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.105695 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.106850 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 54 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:40.124362 10314 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250114 03:05:40.125677 10321 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250114 03:05:40.158253 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.159420 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 55 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.211097 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.212399 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 56 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.263825 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.265074 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 57 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.316578 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.317833 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 58 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.369496 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.370775 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 59 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.422236 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.423592 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 60 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:40.441478 10173 consensus_peers.cc:487] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 -> Peer b2b948e36095485faa6060007121083a (127.1.109.66:39785): Couldn't send request to peer b2b948e36095485faa6060007121083a. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250114 03:05:40.475208 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.476806 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 61 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.528697 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.530150 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 62 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.581789 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.582957 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 63 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.634336 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.635596 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 64 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.687238 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.688472 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 65 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.739946 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.741238 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 66 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.792601 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.793850 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 67 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.845218 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.846417 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 68 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.897902 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.899027 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 69 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:40.950573 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:40.951800 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 70 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.003252 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.004489 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 71 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.056020 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.057628 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 72 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.109237 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.110535 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 73 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.161893 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.163106 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 74 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.214566 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.215865 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 75 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.267439 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.268649 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 76 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:41.299278 10173 consensus_peers.cc:487] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 -> Peer b2b948e36095485faa6060007121083a (127.1.109.66:39785): Couldn't send request to peer b2b948e36095485faa6060007121083a. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250114 03:05:41.320297 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.321588 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 77 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.325469 10346 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58566 (ReqId={client: 01b740fea92940d0a85b2221e055b441, seq_no=2, attempt_no=0}) took 1202 ms. Trace:
I20250114 03:05:41.325572 10346 rpcz_store.cc:276] 0114 03:05:40.122965 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:40.123053 (+ 88us) service_pool.cc:225] Handling call
0114 03:05:41.325441 (+1202388us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:40.123673 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e9d59cec93c04246b9b519e39015aa59
0114 03:05:40.123769 (+ 96us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:40.123773 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:05:40.123774 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:40.123847 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:40.123855 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:40.123857 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:40.123890 (+ 33us) tablet.cc:698] Row locks acquired
0114 03:05:40.123891 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:40.123923 (+ 32us) write_op.cc:270] Start()
0114 03:05:40.123947 (+ 24us) write_op.cc:276] Timestamp: P: 1736823940123848 usec, L: 0
0114 03:05:40.123948 (+ 1us) op_driver.cc:348] REPLICATION: starting
0114 03:05:40.124100 (+ 152us) log.cc:844] Serialized 354 byte log entry
0114 03:05:41.324812 (+1200712us) op_driver.cc:464] REPLICATION: finished
0114 03:05:41.325052 (+ 240us) write_op.cc:301] APPLY: starting
0114 03:05:41.325070 (+ 18us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:41.325100 (+ 30us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:41.325114 (+ 14us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:41.325166 (+ 52us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:41.325189 (+ 23us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:41.325192 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:05:41.325251 (+ 59us) log.cc:844] Serialized 22 byte log entry
0114 03:05:41.325331 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:41.325348 (+ 17us) write_op.cc:454] Released schema lock
0114 03:05:41.325354 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":163,"num_ops":1,"prepare.queue_time_us":199,"prepare.run_cpu_time_us":541,"prepare.run_wall_time_us":541,"raft.queue_time_us":87,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":199,"replication_time_us":1200826,"spinlock_wait_cycles":10112,"thread_start_us":183,"threads_started":2}]]}
I20250114 03:05:41.373165 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.374409 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 78 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.425855 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.427032 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 79 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.478583 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.479759 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 80 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:41.528766 10314 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8: Injecting 1200ms of latency in SegmentAllocator::Sync()
W20250114 03:05:41.530143 10321 log.cc:510] T e9d59cec93c04246b9b519e39015aa59 P 3a9a4ca552714a1abc8aba425b3420a9: Injecting 600ms of latency in SegmentAllocator::Sync()
I20250114 03:05:41.530937 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.532095 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 81 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.583626 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.584789 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 82 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.636307 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.637625 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 83 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.689267 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.690434 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 84 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.741838 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.743000 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 85 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.794533 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.795708 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 86 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.847210 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.848435 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 87 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.899839 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.901114 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 88 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:41.952662 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:41.953883 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 89 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.005403 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.006582 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 90 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.057992 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.059248 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 91 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.110832 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.112063 9977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 92 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.163587 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.164865 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 93 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.216490 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.217599 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 94 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.269038 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.270232 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 95 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.321758 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.322957 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 96 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.374531 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.375780 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 97 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
W20250114 03:05:42.391518 10173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20250114 03:05:42.392129 10173 consensus_peers.cc:487] T e9d59cec93c04246b9b519e39015aa59 P bb2e127fa46647c6b868bc92d3fbc9e8 -> Peer b2b948e36095485faa6060007121083a (127.1.109.66:39785): Couldn't send request to peer b2b948e36095485faa6060007121083a. Status: Network error: Client connection negotiation failed: client connection to 127.1.109.66:39785: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20250114 03:05:42.427214 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.428385 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 98 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.480021 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.481467 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 99 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.533074 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.534322 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 100 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.585937 10240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "bb2e127fa46647c6b868bc92d3fbc9e8" is_pre_election: false
I20250114 03:05:42.587152 9978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9d59cec93c04246b9b519e39015aa59" candidate_uuid: "b2b948e36095485faa6060007121083a" candidate_term: 2 candidate_status { last_received { term: 101 index: 0 } } ignore_live_leader: false dest_uuid: "3a9a4ca552714a1abc8aba425b3420a9" is_pre_election: false
I20250114 03:05:42.730114 10352 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.Write from 127.0.0.1:58566 (ReqId={client: 01b740fea92940d0a85b2221e055b441, seq_no=3, attempt_no=0}) took 1202 ms. Trace:
I20250114 03:05:42.730218 10352 rpcz_store.cc:276] 0114 03:05:41.527313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0114 03:05:41.527399 (+ 86us) service_pool.cc:225] Handling call
0114 03:05:42.730091 (+1202692us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0114 03:05:41.528089 (+ 0us) write_op.cc:183] PREPARE: starting on tablet e9d59cec93c04246b9b519e39015aa59
0114 03:05:41.528183 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0114 03:05:41.528187 (+ 4us) write_op.cc:435] Acquired schema lock
0114 03:05:41.528188 (+ 1us) tablet.cc:659] Decoding operations
0114 03:05:41.528253 (+ 65us) write_op.cc:620] Acquiring the partition lock for write op
0114 03:05:41.528262 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0114 03:05:41.528264 (+ 2us) tablet.cc:682] Acquiring locks for 1 operations
0114 03:05:41.528298 (+ 34us) tablet.cc:698] Row locks acquired
0114 03:05:41.528299 (+ 1us) write_op.cc:260] PREPARE: finished
0114 03:05:41.528330 (+ 31us) write_op.cc:270] Start()
0114 03:05:41.528356 (+ 26us) write_op.cc:276] Timestamp: P: 1736823941528267 usec, L: 0
0114 03:05:41.528357 (+ 1us) op_driver.cc:348] REPLICATION: starting
0114 03:05:41.528507 (+ 150us) log.cc:844] Serialized 354 byte log entry
0114 03:05:42.729366 (+1200859us) op_driver.cc:464] REPLICATION: finished
0114 03:05:42.729650 (+ 284us) write_op.cc:301] APPLY: starting
0114 03:05:42.729671 (+ 21us) tablet.cc:1364] starting BulkCheckPresence
0114 03:05:42.729720 (+ 49us) tablet.cc:1367] finished BulkCheckPresence
0114 03:05:42.729745 (+ 25us) tablet.cc:1369] starting ApplyRowOperation cycle
0114 03:05:42.729810 (+ 65us) tablet.cc:1380] finished ApplyRowOperation cycle
0114 03:05:42.729843 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0114 03:05:42.729846 (+ 3us) write_op.cc:312] APPLY: finished
0114 03:05:42.729911 (+ 65us) log.cc:844] Serialized 22 byte log entry
0114 03:05:42.729972 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0114 03:05:42.729989 (+ 17us) write_op.cc:454] Released schema lock
0114 03:05:42.729994 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":215,"num_ops":1,"prepare.queue_time_us":213,"prepare.run_cpu_time_us":532,"prepare.run_wall_time_us":533,"raft.queue_time_us":184,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":260,"replication_time_us":1200959,"thread_start_us":189,"threads_started":2}]]}
I20250114 03:05:42.932126 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 9896
I20250114 03:05:42.939069 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10158
I20250114 03:05:42.945515 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 9805
2025-01-14T03:05:42Z chronyd exiting
[ OK ] RaftConsensusElectionITest.DisruptiveServerAndSlowWAL (6746 ms)
[----------] 12 tests from RaftConsensusElectionITest (170587 ms total)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0
I20250114 03:05:42.968078 1461 test_util.cc:274] Using random seed: -701992185
I20250114 03:05:42.968951 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:42.968999 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:42.969018 1461 ts_itest-base.cc:114] 1 tablet servers
I20250114 03:05:42.969036 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:05:42.969061 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:42Z Disabled control of system clock
I20250114 03:05:42.977653 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:32807
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:32807
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250114 03:05:43.087909 10363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:43.088165 10363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:43.088241 10363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:43.091398 10363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:43.091466 10363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:43.091498 10363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:43.091542 10363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:43.095366 10363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:32807
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:32807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:43.096482 10363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:43.097733 10363 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:43.105746 10369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:43.105957 10368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:43.106750 10371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:43.107602 10363 server_base.cc:1034] running on GCE node
I20250114 03:05:43.108094 10363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:43.109117 10363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:43.110262 10363 hybrid_clock.cc:648] HybridClock initialized: now 1736823943110246 us; error 36 us; skew 500 ppm
I20250114 03:05:43.116199 10363 webserver.cc:458] Webserver started at http://127.1.109.126:38841/ using document root <none> and password file <none>
I20250114 03:05:43.116914 10363 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:43.116977 10363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:43.117182 10363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:43.118937 10363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "966005bae1a5461d9fb50219caace889"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.119428 10363 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "966005bae1a5461d9fb50219caace889"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.122920 10363 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20250114 03:05:43.125037 10377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:43.126089 10363 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:43.126204 10363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "966005bae1a5461d9fb50219caace889"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.126291 10363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:43.150838 10363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:43.151530 10363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:43.151697 10363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:43.158528 10363 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:32807
I20250114 03:05:43.158553 10429 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:32807 every 8 connection(s)
I20250114 03:05:43.159590 10363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:43.162700 10430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.163342 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10363
I20250114 03:05:43.163471 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:43.167870 10430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889: Bootstrap starting.
I20250114 03:05:43.170317 10430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.171131 10430 log.cc:826] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:43.172931 10430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889: No bootstrap required, opened a new log
I20250114 03:05:43.176059 10430 raft_consensus.cc:357] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966005bae1a5461d9fb50219caace889" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 32807 } }
I20250114 03:05:43.176250 10430 raft_consensus.cc:383] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:43.176303 10430 raft_consensus.cc:738] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966005bae1a5461d9fb50219caace889, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.176720 10430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966005bae1a5461d9fb50219caace889" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 32807 } }
I20250114 03:05:43.176867 10430 raft_consensus.cc:397] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:43.176929 10430 raft_consensus.cc:491] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:43.177040 10430 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.177824 10430 raft_consensus.cc:513] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966005bae1a5461d9fb50219caace889" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 32807 } }
I20250114 03:05:43.178161 10430 leader_election.cc:304] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 966005bae1a5461d9fb50219caace889; no voters:
I20250114 03:05:43.178438 10430 leader_election.cc:290] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.178550 10435 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.178794 10435 raft_consensus.cc:695] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [term 1 LEADER]: Becoming Leader. State: Replica: 966005bae1a5461d9fb50219caace889, State: Running, Role: LEADER
I20250114 03:05:43.179163 10435 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966005bae1a5461d9fb50219caace889" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 32807 } }
I20250114 03:05:43.179600 10430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:43.180757 10436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "966005bae1a5461d9fb50219caace889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966005bae1a5461d9fb50219caace889" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 32807 } } }
I20250114 03:05:43.180761 10437 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966005bae1a5461d9fb50219caace889. Latest consensus state: current_term: 1 leader_uuid: "966005bae1a5461d9fb50219caace889" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "966005bae1a5461d9fb50219caace889" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 32807 } } }
I20250114 03:05:43.180924 10436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:43.180924 10437 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:43.181329 10444 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:43.184137 10444 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:43.189864 10444 catalog_manager.cc:1348] Generated new cluster ID: b43d8fab3646488fa688e94aec2ae660
I20250114 03:05:43.189944 10444 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:43.195043 10444 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:43.196288 10444 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:43.204476 10444 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 966005bae1a5461d9fb50219caace889: Generated new TSK 0
I20250114 03:05:43.205464 10444 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:43.208734 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:32807
--builtin_ntp_servers=127.1.109.84:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250114 03:05:43.319167 10454 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:43.319427 10454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:43.319480 10454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:43.319553 10454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:43.322711 10454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:43.322865 10454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:43.326783 10454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:32807
--never_fsync=true
--heap_profile_path=/tmp/kudu.10454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:43.327966 10454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:43.329205 10454 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:43.338801 10460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:43.339046 10459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:43.339679 10462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:43.340668 10454 server_base.cc:1034] running on GCE node
I20250114 03:05:43.341192 10454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:43.341847 10454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:43.343019 10454 hybrid_clock.cc:648] HybridClock initialized: now 1736823943342993 us; error 47 us; skew 500 ppm
I20250114 03:05:43.345206 10454 webserver.cc:458] Webserver started at http://127.1.109.65:45557/ using document root <none> and password file <none>
I20250114 03:05:43.345819 10454 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:43.345880 10454 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:43.346068 10454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:43.347797 10454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d80a6f5b725b4297b5201940a6ab2265"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.348309 10454 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d80a6f5b725b4297b5201940a6ab2265"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.351819 10454 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:05:43.354259 10468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:43.355401 10454 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:43.355527 10454 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d80a6f5b725b4297b5201940a6ab2265"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.355640 10454 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:43.365902 10454 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:43.366544 10454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:43.366708 10454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:43.367220 10454 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:43.368091 10454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:43.368136 10454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:43.368182 10454 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:43.368213 10454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:43.376977 10454 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:41711
I20250114 03:05:43.377013 10581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:41711 every 8 connection(s)
I20250114 03:05:43.377974 10454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:43.384706 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10454
I20250114 03:05:43.384851 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:43.386924 10582 heartbeater.cc:346] Connected to a master server at 127.1.109.126:32807
I20250114 03:05:43.387156 10582 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:43.387727 10582 heartbeater.cc:510] Master 127.1.109.126:32807 requested a full tablet report, sending...
I20250114 03:05:43.389189 10394 ts_manager.cc:194] Registered new tserver with Master: d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65:41711)
I20250114 03:05:43.390389 10394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:34719
I20250114 03:05:43.399048 1461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250114 03:05:43.408174 1461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250114 03:05:43.408265 1461 test_util.cc:274] Using random seed: -701551992
I20250114 03:05:43.419158 10394 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33756:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250114 03:05:43.437852 10508 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2d5723ccb544598cf1c2987bad9865 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 03:05:43.438302 10513 tablet_service.cc:1467] Processing CreateTablet for tablet ecd24e9c93bc4e56ac14db0fb0df5a9b (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 03:05:43.437717 10511 tablet_service.cc:1467] Processing CreateTablet for tablet ae87ca31064444bcb02190479968d398 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 03:05:43.438627 10514 tablet_service.cc:1467] Processing CreateTablet for tablet 89256ee48cc740369e56b5b821fcb7e8 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 03:05:43.438194 10507 tablet_service.cc:1467] Processing CreateTablet for tablet e8acb106dd134713b6eb27de04d3e763 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 03:05:43.437717 10512 tablet_service.cc:1467] Processing CreateTablet for tablet e455e04efd9d44bab5e26ca8f257a36f (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 03:05:43.438843 10516 tablet_service.cc:1467] Processing CreateTablet for tablet 27181ad73e344160998768a6a8a0dc34 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 03:05:43.437914 10510 tablet_service.cc:1467] Processing CreateTablet for tablet a6d941cf84f14017a9b6fbe82af5387d (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 03:05:43.437726 10509 tablet_service.cc:1467] Processing CreateTablet for tablet 259fb4cee2ff427182b162117d0f7225 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 03:05:43.439074 10515 tablet_service.cc:1467] Processing CreateTablet for tablet 176d48ebf239476fbe00302a79d18924 (DEFAULT_TABLE table=test-workload [id=957dbdcc0ac14abaade9e9175b5e3b17]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 03:05:43.439276 10511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae87ca31064444bcb02190479968d398. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.439723 10510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d941cf84f14017a9b6fbe82af5387d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.439867 10507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e8acb106dd134713b6eb27de04d3e763. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.440013 10516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27181ad73e344160998768a6a8a0dc34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.440979 10508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a2d5723ccb544598cf1c2987bad9865. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.441125 10514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89256ee48cc740369e56b5b821fcb7e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.441262 10513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecd24e9c93bc4e56ac14db0fb0df5a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.441589 10512 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e455e04efd9d44bab5e26ca8f257a36f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.442126 10515 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 176d48ebf239476fbe00302a79d18924. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.448042 10509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 259fb4cee2ff427182b162117d0f7225. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.450250 10597 tablet_bootstrap.cc:492] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.452204 10597 tablet_bootstrap.cc:654] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.452945 10597 log.cc:826] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:43.454524 10597 tablet_bootstrap.cc:492] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.454680 10597 ts_tablet_manager.cc:1397] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20250114 03:05:43.457804 10597 raft_consensus.cc:357] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.457993 10597 raft_consensus.cc:738] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.458256 10597 consensus_queue.cc:260] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.458932 10582 heartbeater.cc:502] Master 127.1.109.126:32807 was elected leader, sending a full tablet report...
I20250114 03:05:43.459100 10597 ts_tablet_manager.cc:1428] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.004s user 0.000s sys 0.005s
I20250114 03:05:43.459362 10597 tablet_bootstrap.cc:492] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.460502 10597 tablet_bootstrap.cc:654] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.461652 10597 tablet_bootstrap.cc:492] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.461728 10597 ts_tablet_manager.cc:1397] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250114 03:05:43.462193 10597 raft_consensus.cc:357] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.462293 10597 raft_consensus.cc:738] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.462397 10597 consensus_queue.cc:260] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.462699 10597 ts_tablet_manager.cc:1428] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250114 03:05:43.462862 10597 tablet_bootstrap.cc:492] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.463835 10597 tablet_bootstrap.cc:654] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.465162 10597 tablet_bootstrap.cc:492] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.465279 10597 ts_tablet_manager.cc:1397] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20250114 03:05:43.465773 10597 raft_consensus.cc:357] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.465871 10597 raft_consensus.cc:738] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.465968 10597 consensus_queue.cc:260] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.466233 10597 ts_tablet_manager.cc:1428] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:43.466408 10597 tablet_bootstrap.cc:492] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.467379 10597 tablet_bootstrap.cc:654] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.468972 10597 tablet_bootstrap.cc:492] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.469074 10597 ts_tablet_manager.cc:1397] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:43.469542 10597 raft_consensus.cc:357] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.469652 10597 raft_consensus.cc:738] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.469756 10597 consensus_queue.cc:260] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.470065 10597 ts_tablet_manager.cc:1428] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:43.470250 10597 tablet_bootstrap.cc:492] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.471268 10597 tablet_bootstrap.cc:654] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.472347 10597 tablet_bootstrap.cc:492] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.472429 10597 ts_tablet_manager.cc:1397] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:43.472851 10597 raft_consensus.cc:357] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.472962 10597 raft_consensus.cc:738] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.473067 10597 consensus_queue.cc:260] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.473325 10597 ts_tablet_manager.cc:1428] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:43.473516 10597 tablet_bootstrap.cc:492] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.474421 10597 tablet_bootstrap.cc:654] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.475443 10597 tablet_bootstrap.cc:492] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.475545 10597 ts_tablet_manager.cc:1397] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:43.475975 10597 raft_consensus.cc:357] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.476083 10597 raft_consensus.cc:738] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.476181 10597 consensus_queue.cc:260] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.476424 10597 ts_tablet_manager.cc:1428] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:43.476603 10597 tablet_bootstrap.cc:492] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.477667 10597 tablet_bootstrap.cc:654] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.478945 10597 tablet_bootstrap.cc:492] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.479038 10597 ts_tablet_manager.cc:1397] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:43.479548 10597 raft_consensus.cc:357] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.479640 10597 raft_consensus.cc:738] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.479724 10597 consensus_queue.cc:260] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.480082 10597 ts_tablet_manager.cc:1428] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:43.480263 10597 tablet_bootstrap.cc:492] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.481271 10597 tablet_bootstrap.cc:654] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.482542 10597 tablet_bootstrap.cc:492] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.482628 10597 ts_tablet_manager.cc:1397] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:43.483072 10597 raft_consensus.cc:357] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.483167 10597 raft_consensus.cc:738] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.483276 10597 consensus_queue.cc:260] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.483551 10597 ts_tablet_manager.cc:1428] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:43.483742 10597 tablet_bootstrap.cc:492] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.484584 10597 tablet_bootstrap.cc:654] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.485777 10597 tablet_bootstrap.cc:492] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.485860 10597 ts_tablet_manager.cc:1397] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:43.486289 10597 raft_consensus.cc:357] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.486397 10597 raft_consensus.cc:738] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.486497 10597 consensus_queue.cc:260] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.486796 10597 ts_tablet_manager.cc:1428] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:43.486969 10597 tablet_bootstrap.cc:492] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: Bootstrap starting.
I20250114 03:05:43.488030 10597 tablet_bootstrap.cc:654] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.489166 10597 tablet_bootstrap.cc:492] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: No bootstrap required, opened a new log
I20250114 03:05:43.489256 10597 ts_tablet_manager.cc:1397] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:43.489706 10597 raft_consensus.cc:357] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.489799 10597 raft_consensus.cc:738] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.489876 10597 consensus_queue.cc:260] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.490139 10597 ts_tablet_manager.cc:1428] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:43.515434 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "259fb4cee2ff427182b162117d0f7225"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.515708 10536 raft_consensus.cc:491] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.515825 10536 raft_consensus.cc:3054] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.516858 10536 raft_consensus.cc:513] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.517248 10536 leader_election.cc:304] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.517473 10536 leader_election.cc:290] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.517546 10599 raft_consensus.cc:2798] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.517807 10599 raft_consensus.cc:695] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.518157 10599 consensus_queue.cc:237] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.521153 10394 catalog_manager.cc:5526] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.526501 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "27181ad73e344160998768a6a8a0dc34"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.526698 10536 raft_consensus.cc:491] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.526767 10536 raft_consensus.cc:3054] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.527580 10536 raft_consensus.cc:513] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.527737 10536 leader_election.cc:304] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.527822 10536 leader_election.cc:290] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.527896 10599 raft_consensus.cc:2798] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.528034 10599 raft_consensus.cc:695] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.528195 10599 consensus_queue.cc:237] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.529903 10394 catalog_manager.cc:5526] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.535562 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a2d5723ccb544598cf1c2987bad9865"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.535729 10536 raft_consensus.cc:491] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.535796 10536 raft_consensus.cc:3054] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.536595 10536 raft_consensus.cc:513] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.536743 10536 leader_election.cc:304] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.536870 10536 leader_election.cc:290] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.536973 10599 raft_consensus.cc:2798] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.537092 10599 raft_consensus.cc:695] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.537240 10599 consensus_queue.cc:237] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.539120 10394 catalog_manager.cc:5526] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.544665 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "176d48ebf239476fbe00302a79d18924"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.544857 10536 raft_consensus.cc:491] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.544934 10536 raft_consensus.cc:3054] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.545751 10536 raft_consensus.cc:513] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.545897 10536 leader_election.cc:304] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.546011 10536 leader_election.cc:290] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.546077 10599 raft_consensus.cc:2798] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.546198 10599 raft_consensus.cc:695] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.546376 10599 consensus_queue.cc:237] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.548077 10394 catalog_manager.cc:5526] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.552984 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89256ee48cc740369e56b5b821fcb7e8"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.553128 10536 raft_consensus.cc:491] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.553175 10536 raft_consensus.cc:3054] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.554050 10536 raft_consensus.cc:513] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.554189 10536 leader_election.cc:304] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.554271 10536 leader_election.cc:290] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.554364 10599 raft_consensus.cc:2798] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.554451 10599 raft_consensus.cc:695] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.554582 10599 consensus_queue.cc:237] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.556185 10394 catalog_manager.cc:5526] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.561419 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a6d941cf84f14017a9b6fbe82af5387d"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.561576 10536 raft_consensus.cc:491] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.561643 10536 raft_consensus.cc:3054] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.562430 10536 raft_consensus.cc:513] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.562575 10536 leader_election.cc:304] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.562661 10536 leader_election.cc:290] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.562734 10599 raft_consensus.cc:2798] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.562855 10599 raft_consensus.cc:695] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.562997 10599 consensus_queue.cc:237] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.564718 10394 catalog_manager.cc:5526] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.569826 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e455e04efd9d44bab5e26ca8f257a36f"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.569964 10536 raft_consensus.cc:491] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.570012 10536 raft_consensus.cc:3054] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.570744 10536 raft_consensus.cc:513] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.570873 10536 leader_election.cc:304] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.570961 10536 leader_election.cc:290] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.571031 10599 raft_consensus.cc:2798] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.571139 10599 raft_consensus.cc:695] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.571267 10599 consensus_queue.cc:237] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.572921 10394 catalog_manager.cc:5526] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.577821 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e8acb106dd134713b6eb27de04d3e763"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.577971 10536 raft_consensus.cc:491] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.578018 10536 raft_consensus.cc:3054] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.578758 10536 raft_consensus.cc:513] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.578886 10536 leader_election.cc:304] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.578966 10536 leader_election.cc:290] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.579030 10599 raft_consensus.cc:2798] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.579154 10599 raft_consensus.cc:695] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.579291 10599 consensus_queue.cc:237] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.580961 10394 catalog_manager.cc:5526] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.585666 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ae87ca31064444bcb02190479968d398"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.585816 10536 raft_consensus.cc:491] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.585886 10536 raft_consensus.cc:3054] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.586665 10536 raft_consensus.cc:513] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.586813 10536 leader_election.cc:304] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.586926 10536 leader_election.cc:290] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.587028 10599 raft_consensus.cc:2798] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.587145 10599 raft_consensus.cc:695] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.587287 10599 consensus_queue.cc:237] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.588917 10394 catalog_manager.cc:5526] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.593922 10536 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ecd24e9c93bc4e56ac14db0fb0df5a9b"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.594069 10536 raft_consensus.cc:491] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:43.594135 10536 raft_consensus.cc:3054] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.594908 10536 raft_consensus.cc:513] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.595048 10536 leader_election.cc:304] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d80a6f5b725b4297b5201940a6ab2265; no voters:
I20250114 03:05:43.595156 10536 leader_election.cc:290] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.595223 10599 raft_consensus.cc:2798] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.595435 10599 raft_consensus.cc:695] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Leader. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.595614 10599 consensus_queue.cc:237] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.597149 10394 catalog_manager.cc:5526] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 reported cstate change: term changed from 0 to 1, leader changed from <none> to d80a6f5b725b4297b5201940a6ab2265 (127.1.109.65). New cstate: current_term: 1 leader_uuid: "d80a6f5b725b4297b5201940a6ab2265" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:43.602131 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet 259fb4cee2ff427182b162117d0f7225 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.602624 10623 tablet_replica.cc:331] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.602926 10623 raft_consensus.cc:2238] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:43.603138 10623 raft_consensus.cc:2267] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.604338 10623 ts_tablet_manager.cc:1905] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.606357 10623 ts_tablet_manager.cc:1918] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.606434 10623 log.cc:1198] T 259fb4cee2ff427182b162117d0f7225 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/259fb4cee2ff427182b162117d0f7225
I20250114 03:05:43.612361 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet 27181ad73e344160998768a6a8a0dc34 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.612615 10623 tablet_replica.cc:331] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.612772 10623 raft_consensus.cc:2238] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:43.612939 10623 raft_consensus.cc:2267] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.613646 10623 ts_tablet_manager.cc:1905] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.615422 10623 ts_tablet_manager.cc:1918] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.615484 10623 log.cc:1198] T 27181ad73e344160998768a6a8a0dc34 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/27181ad73e344160998768a6a8a0dc34
I20250114 03:05:43.621254 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2d5723ccb544598cf1c2987bad9865 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.621471 10623 tablet_replica.cc:331] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.621626 10623 raft_consensus.cc:2238] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:43.621753 10623 raft_consensus.cc:2267] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.622390 10623 ts_tablet_manager.cc:1905] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.624086 10623 ts_tablet_manager.cc:1918] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.624147 10623 log.cc:1198] T 9a2d5723ccb544598cf1c2987bad9865 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a2d5723ccb544598cf1c2987bad9865
I20250114 03:05:43.630138 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet 176d48ebf239476fbe00302a79d18924 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.630390 10623 tablet_replica.cc:331] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.630538 10623 raft_consensus.cc:2238] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:43.630667 10623 raft_consensus.cc:2267] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.631340 10623 ts_tablet_manager.cc:1905] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.633152 10623 ts_tablet_manager.cc:1918] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.633229 10623 log.cc:1198] T 176d48ebf239476fbe00302a79d18924 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/176d48ebf239476fbe00302a79d18924
I20250114 03:05:43.639699 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet 89256ee48cc740369e56b5b821fcb7e8 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.639928 10623 tablet_replica.cc:331] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.640045 10623 raft_consensus.cc:2238] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:43.640162 10623 raft_consensus.cc:2267] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.640914 10623 ts_tablet_manager.cc:1905] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.642632 10623 ts_tablet_manager.cc:1918] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.642702 10623 log.cc:1198] T 89256ee48cc740369e56b5b821fcb7e8 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/89256ee48cc740369e56b5b821fcb7e8
I20250114 03:05:43.648455 10536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a6d941cf84f14017a9b6fbe82af5387d"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.648672 10536 raft_consensus.cc:589] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Received request to step down
I20250114 03:05:43.648723 10536 raft_consensus.cc:3049] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:43.648772 10536 raft_consensus.cc:738] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.648947 10536 consensus_queue.cc:260] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.649029 10536 raft_consensus.cc:3054] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:43.654430 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet a6d941cf84f14017a9b6fbe82af5387d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.654642 10623 tablet_replica.cc:331] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.654759 10623 raft_consensus.cc:2238] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:43.654866 10623 raft_consensus.cc:2267] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.655505 10623 ts_tablet_manager.cc:1905] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.657202 10623 ts_tablet_manager.cc:1918] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.657264 10623 log.cc:1198] T a6d941cf84f14017a9b6fbe82af5387d P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/a6d941cf84f14017a9b6fbe82af5387d
I20250114 03:05:43.662981 10536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e455e04efd9d44bab5e26ca8f257a36f"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.663151 10536 raft_consensus.cc:589] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Received request to step down
I20250114 03:05:43.663213 10536 raft_consensus.cc:3049] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:43.663250 10536 raft_consensus.cc:738] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.663386 10536 consensus_queue.cc:260] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.663462 10536 raft_consensus.cc:3054] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:43.668802 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet e455e04efd9d44bab5e26ca8f257a36f with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.669063 10623 tablet_replica.cc:331] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.669211 10623 raft_consensus.cc:2238] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:43.669337 10623 raft_consensus.cc:2267] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.669978 10623 ts_tablet_manager.cc:1905] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.671799 10623 ts_tablet_manager.cc:1918] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.671890 10623 log.cc:1198] T e455e04efd9d44bab5e26ca8f257a36f P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/e455e04efd9d44bab5e26ca8f257a36f
I20250114 03:05:43.677886 10536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e8acb106dd134713b6eb27de04d3e763"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.678081 10536 raft_consensus.cc:589] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Received request to step down
I20250114 03:05:43.678136 10536 raft_consensus.cc:3049] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:43.678179 10536 raft_consensus.cc:738] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.678323 10536 consensus_queue.cc:260] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.678402 10536 raft_consensus.cc:3054] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:43.684239 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet e8acb106dd134713b6eb27de04d3e763 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.684468 10623 tablet_replica.cc:331] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.684612 10623 raft_consensus.cc:2238] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:43.684747 10623 raft_consensus.cc:2267] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.685489 10623 ts_tablet_manager.cc:1905] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.687410 10623 ts_tablet_manager.cc:1918] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.687484 10623 log.cc:1198] T e8acb106dd134713b6eb27de04d3e763 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/e8acb106dd134713b6eb27de04d3e763
I20250114 03:05:43.693256 10536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ae87ca31064444bcb02190479968d398"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.693416 10536 raft_consensus.cc:589] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Received request to step down
I20250114 03:05:43.693483 10536 raft_consensus.cc:3049] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:43.693518 10536 raft_consensus.cc:738] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.693646 10536 consensus_queue.cc:260] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.693720 10536 raft_consensus.cc:3054] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:43.699107 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet ae87ca31064444bcb02190479968d398 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.699321 10623 tablet_replica.cc:331] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.699432 10623 raft_consensus.cc:2238] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:43.699539 10623 raft_consensus.cc:2267] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.700204 10623 ts_tablet_manager.cc:1905] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.702004 10623 ts_tablet_manager.cc:1918] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.702069 10623 log.cc:1198] T ae87ca31064444bcb02190479968d398 P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/ae87ca31064444bcb02190479968d398
I20250114 03:05:43.709636 10536 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ecd24e9c93bc4e56ac14db0fb0df5a9b"
dest_uuid: "d80a6f5b725b4297b5201940a6ab2265"
from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.709823 10536 raft_consensus.cc:589] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Received request to step down
I20250114 03:05:43.709879 10536 raft_consensus.cc:3049] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:43.709911 10536 raft_consensus.cc:738] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d80a6f5b725b4297b5201940a6ab2265, State: Running, Role: LEADER
I20250114 03:05:43.710040 10536 consensus_queue.cc:260] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d80a6f5b725b4297b5201940a6ab2265" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41711 } }
I20250114 03:05:43.710114 10536 raft_consensus.cc:3054] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:43.716593 10509 tablet_service.cc:1514] Processing DeleteTablet for tablet ecd24e9c93bc4e56ac14db0fb0df5a9b with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:51826
I20250114 03:05:43.716892 10623 tablet_replica.cc:331] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: stopping tablet replica
I20250114 03:05:43.717046 10623 raft_consensus.cc:2238] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:43.717180 10623 raft_consensus.cc:2267] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:43.717926 10623 ts_tablet_manager.cc:1905] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:43.719784 10623 ts_tablet_manager.cc:1918] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:43.719851 10623 log.cc:1198] T ecd24e9c93bc4e56ac14db0fb0df5a9b P d80a6f5b725b4297b5201940a6ab2265: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_0.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/wals/ecd24e9c93bc4e56ac14db0fb0df5a9b
I20250114 03:05:43.726603 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10454
I20250114 03:05:43.733907 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10363
2025-01-14T03:05:43Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/0 (787 ms)
[ RUN ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1
I20250114 03:05:43.755257 1461 test_util.cc:274] Using random seed: -701205005
I20250114 03:05:43.756001 1461 ts_itest-base.cc:112] Starting cluster with:
I20250114 03:05:43.756048 1461 ts_itest-base.cc:113] --------------
I20250114 03:05:43.756060 1461 ts_itest-base.cc:114] 3 tablet servers
I20250114 03:05:43.756071 1461 ts_itest-base.cc:115] 3 replicas per TS
I20250114 03:05:43.756081 1461 ts_itest-base.cc:116] --------------
2025-01-14T03:05:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-14T03:05:43Z Disabled control of system clock
I20250114 03:05:43.764497 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:40153
--webserver_interface=127.1.109.126
--webserver_port=0
--builtin_ntp_servers=127.1.109.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.1.109.126:40153
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250114 03:05:43.874713 10631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:43.874982 10631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:43.875048 10631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:43.878327 10631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250114 03:05:43.878405 10631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:43.878446 10631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250114 03:05:43.878489 10631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250114 03:05:43.882462 10631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.1.109.126:40153
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.1.109.126:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.1.109.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.10631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:43.883661 10631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:43.884871 10631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:43.892735 10639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:43.893033 10636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:43.893857 10637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:43.894886 10631 server_base.cc:1034] running on GCE node
I20250114 03:05:43.895512 10631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:43.896539 10631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:43.897701 10631 hybrid_clock.cc:648] HybridClock initialized: now 1736823943897668 us; error 56 us; skew 500 ppm
I20250114 03:05:43.899787 10631 webserver.cc:458] Webserver started at http://127.1.109.126:34553/ using document root <none> and password file <none>
I20250114 03:05:43.900385 10631 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:43.900446 10631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:43.900632 10631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:43.902415 10631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "96c5787823094808bfa5171c5455aa76"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.902906 10631 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "96c5787823094808bfa5171c5455aa76"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.906345 10631 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20250114 03:05:43.908550 10645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:43.909613 10631 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20250114 03:05:43.909739 10631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "96c5787823094808bfa5171c5455aa76"
format_stamp: "Formatted at 2025-01-14 03:05:43 on dist-test-slave-r16g"
I20250114 03:05:43.909835 10631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:43.925215 10631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:43.925908 10631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:43.926083 10631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:43.932684 10631 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.126:40153
I20250114 03:05:43.932693 10697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.126:40153 every 8 connection(s)
I20250114 03:05:43.933746 10631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250114 03:05:43.936749 10698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:43.940202 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10631
I20250114 03:05:43.940330 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250114 03:05:43.942170 10698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76: Bootstrap starting.
I20250114 03:05:43.944361 10698 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:43.945142 10698 log.cc:826] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:43.946931 10698 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76: No bootstrap required, opened a new log
I20250114 03:05:43.950402 10698 raft_consensus.cc:357] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5787823094808bfa5171c5455aa76" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 40153 } }
I20250114 03:05:43.950610 10698 raft_consensus.cc:383] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250114 03:05:43.950678 10698 raft_consensus.cc:738] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96c5787823094808bfa5171c5455aa76, State: Initialized, Role: FOLLOWER
I20250114 03:05:43.951090 10698 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5787823094808bfa5171c5455aa76" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 40153 } }
I20250114 03:05:43.951221 10698 raft_consensus.cc:397] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250114 03:05:43.951288 10698 raft_consensus.cc:491] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250114 03:05:43.951402 10698 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:43.952226 10698 raft_consensus.cc:513] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5787823094808bfa5171c5455aa76" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 40153 } }
I20250114 03:05:43.952566 10698 leader_election.cc:304] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96c5787823094808bfa5171c5455aa76; no voters:
I20250114 03:05:43.952886 10698 leader_election.cc:290] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250114 03:05:43.953029 10703 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:43.953253 10703 raft_consensus.cc:695] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [term 1 LEADER]: Becoming Leader. State: Replica: 96c5787823094808bfa5171c5455aa76, State: Running, Role: LEADER
I20250114 03:05:43.953590 10703 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5787823094808bfa5171c5455aa76" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 40153 } }
I20250114 03:05:43.953992 10698 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [sys.catalog]: configured and running, proceeding with master startup.
I20250114 03:05:43.955228 10705 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96c5787823094808bfa5171c5455aa76. Latest consensus state: current_term: 1 leader_uuid: "96c5787823094808bfa5171c5455aa76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5787823094808bfa5171c5455aa76" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 40153 } } }
I20250114 03:05:43.955367 10705 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:43.955215 10704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96c5787823094808bfa5171c5455aa76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96c5787823094808bfa5171c5455aa76" member_type: VOTER last_known_addr { host: "127.1.109.126" port: 40153 } } }
I20250114 03:05:43.955605 10704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76 [sys.catalog]: This master's current role is: LEADER
I20250114 03:05:43.955850 10712 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250114 03:05:43.958750 10712 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250114 03:05:43.964390 10712 catalog_manager.cc:1348] Generated new cluster ID: a02eb4df3d2548d1b2c0fa893b2496ec
I20250114 03:05:43.964491 10712 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250114 03:05:43.983350 10712 catalog_manager.cc:1371] Generated new certificate authority record
I20250114 03:05:43.984665 10712 catalog_manager.cc:1505] Loading token signing keys...
I20250114 03:05:43.992513 10712 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 96c5787823094808bfa5171c5455aa76: Generated new TSK 0
I20250114 03:05:43.993463 10712 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250114 03:05:43.997150 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.65:0
--local_ip_for_outbound_sockets=127.1.109.65
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:40153
--builtin_ntp_servers=127.1.109.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250114 03:05:44.108420 10722 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:44.108677 10722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:44.108726 10722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:44.108811 10722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:44.112025 10722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:44.112185 10722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.65
I20250114 03:05:44.116134 10722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.1.109.65
--webserver_port=0
--tserver_master_addrs=127.1.109.126:40153
--never_fsync=true
--heap_profile_path=/tmp/kudu.10722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.65
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:44.117312 10722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:44.118492 10722 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:44.126956 10728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:44.127488 10727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:44.128194 10730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:44.129058 10722 server_base.cc:1034] running on GCE node
I20250114 03:05:44.129459 10722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:44.130067 10722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:44.131214 10722 hybrid_clock.cc:648] HybridClock initialized: now 1736823944131199 us; error 37 us; skew 500 ppm
I20250114 03:05:44.133190 10722 webserver.cc:458] Webserver started at http://127.1.109.65:39135/ using document root <none> and password file <none>
I20250114 03:05:44.133774 10722 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:44.133837 10722 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:44.134029 10722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:44.135712 10722 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "cb6c11cd2ea1445e9764341bc17f3768"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.136190 10722 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "cb6c11cd2ea1445e9764341bc17f3768"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.139627 10722 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20250114 03:05:44.141806 10736 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.142803 10722 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:44.142911 10722 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "cb6c11cd2ea1445e9764341bc17f3768"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.143000 10722 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:44.161417 10722 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:44.162125 10722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:44.162303 10722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:44.162879 10722 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:44.163817 10722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:44.163867 10722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.163928 10722 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:44.163960 10722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.173247 10722 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.65:41359
I20250114 03:05:44.173291 10849 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.65:41359 every 8 connection(s)
I20250114 03:05:44.174258 10722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250114 03:05:44.183413 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10722
I20250114 03:05:44.183526 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250114 03:05:44.183604 10850 heartbeater.cc:346] Connected to a master server at 127.1.109.126:40153
I20250114 03:05:44.183893 10850 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:44.184557 10850 heartbeater.cc:510] Master 127.1.109.126:40153 requested a full tablet report, sending...
I20250114 03:05:44.185890 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.66:0
--local_ip_for_outbound_sockets=127.1.109.66
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:40153
--builtin_ntp_servers=127.1.109.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:44.186224 10661 ts_manager.cc:194] Registered new tserver with Master: cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359)
I20250114 03:05:44.187783 10661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.65:52511
W20250114 03:05:44.294224 10853 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:44.294495 10853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:44.294530 10853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:44.294601 10853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:44.297902 10853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:44.298060 10853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.66
I20250114 03:05:44.302130 10853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.1.109.66
--webserver_port=0
--tserver_master_addrs=127.1.109.126:40153
--never_fsync=true
--heap_profile_path=/tmp/kudu.10853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.66
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:44.303316 10853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:44.304529 10853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:44.313167 10861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:44.313762 10858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:44.314623 10859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:44.315411 10853 server_base.cc:1034] running on GCE node
I20250114 03:05:44.315850 10853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:44.316430 10853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:44.317612 10853 hybrid_clock.cc:648] HybridClock initialized: now 1736823944317584 us; error 39 us; skew 500 ppm
I20250114 03:05:44.319618 10853 webserver.cc:458] Webserver started at http://127.1.109.66:34935/ using document root <none> and password file <none>
I20250114 03:05:44.320211 10853 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:44.320266 10853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:44.320436 10853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:44.322191 10853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.322685 10853 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.326293 10853 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:44.328506 10867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.329687 10853 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250114 03:05:44.329802 10853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.329890 10853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:44.344365 10853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:44.345048 10853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:44.345194 10853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:44.345705 10853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:44.346598 10853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:44.346645 10853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.346688 10853 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:44.346710 10853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.355260 10853 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.66:35371
I20250114 03:05:44.355279 10980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.66:35371 every 8 connection(s)
I20250114 03:05:44.356344 10853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250114 03:05:44.362097 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10853
I20250114 03:05:44.362206 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250114 03:05:44.364286 1461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
/tmp/dist-test-taskyEDKFA/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.1.109.67:0
--local_ip_for_outbound_sockets=127.1.109.67
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:40153
--builtin_ntp_servers=127.1.109.84:34993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250114 03:05:44.365383 10981 heartbeater.cc:346] Connected to a master server at 127.1.109.126:40153
I20250114 03:05:44.365674 10981 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:44.366277 10981 heartbeater.cc:510] Master 127.1.109.126:40153 requested a full tablet report, sending...
I20250114 03:05:44.367367 10661 ts_manager.cc:194] Registered new tserver with Master: 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.367941 10661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.66:50613
W20250114 03:05:44.475056 10984 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250114 03:05:44.475306 10984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250114 03:05:44.475329 10984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250114 03:05:44.475391 10984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250114 03:05:44.478622 10984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250114 03:05:44.478766 10984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.1.109.67
I20250114 03:05:44.482709 10984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.1.109.84:34993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.1.109.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.1.109.67
--webserver_port=0
--tserver_master_addrs=127.1.109.126:40153
--never_fsync=true
--heap_profile_path=/tmp/kudu.10984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.1.109.67
--log_dir=/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 622683201dc1b5c1b056d84565ee6cae8964ddd2
build type DEBUG
built by None at 14 Jan 2025 02:43:16 UTC on 24a791456cd2
build id 4165
I20250114 03:05:44.483896 10984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250114 03:05:44.485132 10984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250114 03:05:44.493674 10989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:44.494320 10992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250114 03:05:44.495178 10990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250114 03:05:44.495990 10984 server_base.cc:1034] running on GCE node
I20250114 03:05:44.496424 10984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250114 03:05:44.497079 10984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250114 03:05:44.498230 10984 hybrid_clock.cc:648] HybridClock initialized: now 1736823944498210 us; error 36 us; skew 500 ppm
I20250114 03:05:44.500296 10984 webserver.cc:458] Webserver started at http://127.1.109.67:39401/ using document root <none> and password file <none>
I20250114 03:05:44.500917 10984 fs_manager.cc:362] Metadata directory not provided
I20250114 03:05:44.500989 10984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250114 03:05:44.501179 10984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250114 03:05:44.502873 10984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a5041e64b77742878812e2e2d46ebf7b"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.503350 10984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a5041e64b77742878812e2e2d46ebf7b"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.506795 10984 fs_manager.cc:696] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20250114 03:05:44.509164 10998 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.510226 10984 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20250114 03:05:44.510334 10984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a5041e64b77742878812e2e2d46ebf7b"
format_stamp: "Formatted at 2025-01-14 03:05:44 on dist-test-slave-r16g"
I20250114 03:05:44.510426 10984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250114 03:05:44.565356 10984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250114 03:05:44.566064 10984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250114 03:05:44.566231 10984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250114 03:05:44.566780 10984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250114 03:05:44.567734 10984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250114 03:05:44.567799 10984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.567859 10984 ts_tablet_manager.cc:610] Registered 0 tablets
I20250114 03:05:44.567886 10984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250114 03:05:44.577071 10984 rpc_server.cc:307] RPC server started. Bound to: 127.1.109.67:35995
I20250114 03:05:44.577093 11111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.1.109.67:35995 every 8 connection(s)
I20250114 03:05:44.578136 10984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250114 03:05:44.581084 1461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu as pid 10984
I20250114 03:05:44.581174 1461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250114 03:05:44.587427 11112 heartbeater.cc:346] Connected to a master server at 127.1.109.126:40153
I20250114 03:05:44.587688 11112 heartbeater.cc:463] Registering TS with master...
I20250114 03:05:44.588228 11112 heartbeater.cc:510] Master 127.1.109.126:40153 requested a full tablet report, sending...
I20250114 03:05:44.589248 10661 ts_manager.cc:194] Registered new tserver with Master: a5041e64b77742878812e2e2d46ebf7b (127.1.109.67:35995)
I20250114 03:05:44.589807 10661 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.1.109.67:47981
I20250114 03:05:44.595177 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:44.603998 1461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250114 03:05:44.604056 1461 test_util.cc:274] Using random seed: -700356200
I20250114 03:05:44.613359 10661 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56474:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250114 03:05:44.614909 10661 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250114 03:05:44.635129 10777 tablet_service.cc:1467] Processing CreateTablet for tablet 9837cc4d27ce464e910753a7186e3405 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 03:05:44.635326 11042 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa771d443694da5bb7f11ae022093b4 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 03:05:44.635910 10777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9837cc4d27ce464e910753a7186e3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.636476 11042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa771d443694da5bb7f11ae022093b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.637073 11036 tablet_service.cc:1467] Processing CreateTablet for tablet 9837cc4d27ce464e910753a7186e3405 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 03:05:44.637308 11036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9837cc4d27ce464e910753a7186e3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.636955 10915 tablet_service.cc:1467] Processing CreateTablet for tablet 6537b58b18c141caaa7935e56110a5dd (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 03:05:44.637988 10915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6537b58b18c141caaa7935e56110a5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.641016 11040 tablet_service.cc:1467] Processing CreateTablet for tablet e504fd6dfe5c4670a976d1e6e5bd2e01 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 03:05:44.641094 10909 tablet_service.cc:1467] Processing CreateTablet for tablet e504fd6dfe5c4670a976d1e6e5bd2e01 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 03:05:44.641292 11040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e504fd6dfe5c4670a976d1e6e5bd2e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.641314 10913 tablet_service.cc:1467] Processing CreateTablet for tablet 39a4da5ec6e7480983dad214dff36e0d (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 03:05:44.641407 10909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e504fd6dfe5c4670a976d1e6e5bd2e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.641551 10913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a4da5ec6e7480983dad214dff36e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.641880 10908 tablet_service.cc:1467] Processing CreateTablet for tablet 9837cc4d27ce464e910753a7186e3405 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20250114 03:05:44.642154 10908 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9837cc4d27ce464e910753a7186e3405. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.642803 10775 tablet_service.cc:1467] Processing CreateTablet for tablet f3350d31a6834e808c03cdfabd86ed5a (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 03:05:44.643074 10775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3350d31a6834e808c03cdfabd86ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.643669 11039 tablet_service.cc:1467] Processing CreateTablet for tablet 26040bfbffe047818ba6fb4bb22208c5 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 03:05:44.643921 11039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26040bfbffe047818ba6fb4bb22208c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.644093 10912 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa771d443694da5bb7f11ae022093b4 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 03:05:44.644338 10912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa771d443694da5bb7f11ae022093b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.645372 10907 tablet_service.cc:1467] Processing CreateTablet for tablet 26040bfbffe047818ba6fb4bb22208c5 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 03:05:44.645660 10907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26040bfbffe047818ba6fb4bb22208c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.646121 10780 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 03:05:44.646384 10780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.646375 10911 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 03:05:44.646586 10911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.647668 10906 tablet_service.cc:1467] Processing CreateTablet for tablet f3350d31a6834e808c03cdfabd86ed5a (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 03:05:44.647920 10906 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3350d31a6834e808c03cdfabd86ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.648468 10910 tablet_service.cc:1467] Processing CreateTablet for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 03:05:44.648463 11043 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20250114 03:05:44.648694 10910 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.648695 11043 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.649204 11038 tablet_service.cc:1467] Processing CreateTablet for tablet f3350d31a6834e808c03cdfabd86ed5a (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20250114 03:05:44.649487 11038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3350d31a6834e808c03cdfabd86ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.641108 10914 tablet_service.cc:1467] Processing CreateTablet for tablet a9176121f8f94d32812a73e49717ae7a (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 03:05:44.649679 10914 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9176121f8f94d32812a73e49717ae7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.650009 10776 tablet_service.cc:1467] Processing CreateTablet for tablet 26040bfbffe047818ba6fb4bb22208c5 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20250114 03:05:44.650235 10776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26040bfbffe047818ba6fb4bb22208c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.634603 10784 tablet_service.cc:1467] Processing CreateTablet for tablet 6537b58b18c141caaa7935e56110a5dd (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 03:05:44.651369 10784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6537b58b18c141caaa7935e56110a5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.634817 10778 tablet_service.cc:1467] Processing CreateTablet for tablet e504fd6dfe5c4670a976d1e6e5bd2e01 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20250114 03:05:44.651624 10778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e504fd6dfe5c4670a976d1e6e5bd2e01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.652258 10781 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa771d443694da5bb7f11ae022093b4 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20250114 03:05:44.652493 10781 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fa771d443694da5bb7f11ae022093b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.634606 10779 tablet_service.cc:1467] Processing CreateTablet for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 03:05:44.653221 10779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.654495 11041 tablet_service.cc:1467] Processing CreateTablet for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0 (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20250114 03:05:44.654757 11041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.654994 11134 tablet_bootstrap.cc:492] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.655478 11045 tablet_service.cc:1467] Processing CreateTablet for tablet a9176121f8f94d32812a73e49717ae7a (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 03:05:44.655721 11045 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9176121f8f94d32812a73e49717ae7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.656780 10782 tablet_service.cc:1467] Processing CreateTablet for tablet 39a4da5ec6e7480983dad214dff36e0d (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 03:05:44.657049 10782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a4da5ec6e7480983dad214dff36e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.657008 11044 tablet_service.cc:1467] Processing CreateTablet for tablet 39a4da5ec6e7480983dad214dff36e0d (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20250114 03:05:44.657227 11134 tablet_bootstrap.cc:654] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.657229 11044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39a4da5ec6e7480983dad214dff36e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.657912 11134 log.cc:826] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:44.658924 10783 tablet_service.cc:1467] Processing CreateTablet for tablet a9176121f8f94d32812a73e49717ae7a (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20250114 03:05:44.659051 11046 tablet_service.cc:1467] Processing CreateTablet for tablet 6537b58b18c141caaa7935e56110a5dd (DEFAULT_TABLE table=test-workload [id=824cb9bf9db7488098b34ae477f52f6e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20250114 03:05:44.659180 10783 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9176121f8f94d32812a73e49717ae7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.659312 11046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6537b58b18c141caaa7935e56110a5dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250114 03:05:44.659864 11134 tablet_bootstrap.cc:492] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.660060 11134 ts_tablet_manager.cc:1397] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20250114 03:05:44.663786 11137 tablet_bootstrap.cc:492] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.664956 11134 raft_consensus.cc:357] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.665194 11134 raft_consensus.cc:738] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.665546 11134 consensus_queue.cc:260] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.666177 11135 tablet_bootstrap.cc:492] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.666225 11137 tablet_bootstrap.cc:654] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.667058 11137 log.cc:826] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:44.667093 11134 ts_tablet_manager.cc:1428] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250114 03:05:44.667485 11134 tablet_bootstrap.cc:492] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.667503 10981 heartbeater.cc:502] Master 127.1.109.126:40153 was elected leader, sending a full tablet report...
I20250114 03:05:44.668265 11135 tablet_bootstrap.cc:654] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.668552 11134 tablet_bootstrap.cc:654] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.669095 11135 log.cc:826] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768: Log is configured to *not* fsync() on all Append() calls
I20250114 03:05:44.669140 11137 tablet_bootstrap.cc:492] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.669323 11137 ts_tablet_manager.cc:1397] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20250114 03:05:44.669703 11134 tablet_bootstrap.cc:492] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.669775 11134 ts_tablet_manager.cc:1397] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.670287 11134 raft_consensus.cc:357] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.670406 11134 raft_consensus.cc:738] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.670500 11134 consensus_queue.cc:260] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.670764 11134 ts_tablet_manager.cc:1428] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.670928 11134 tablet_bootstrap.cc:492] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.670997 11135 tablet_bootstrap.cc:492] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.671273 11135 ts_tablet_manager.cc:1397] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20250114 03:05:44.671969 11134 tablet_bootstrap.cc:654] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.673244 11134 tablet_bootstrap.cc:492] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.673321 11134 ts_tablet_manager.cc:1397] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.673744 11134 raft_consensus.cc:357] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.673852 11134 raft_consensus.cc:738] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.673702 11137 raft_consensus.cc:357] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.673956 11137 raft_consensus.cc:738] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.673943 11134 consensus_queue.cc:260] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.674191 11134 ts_tablet_manager.cc:1428] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.674368 11134 tablet_bootstrap.cc:492] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.674368 11137 consensus_queue.cc:260] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.675297 11112 heartbeater.cc:502] Master 127.1.109.126:40153 was elected leader, sending a full tablet report...
I20250114 03:05:44.675392 11134 tablet_bootstrap.cc:654] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.675802 11137 ts_tablet_manager.cc:1428] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250114 03:05:44.676100 11137 tablet_bootstrap.cc:492] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.676982 11134 tablet_bootstrap.cc:492] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.677066 11134 ts_tablet_manager.cc:1397] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.677400 11137 tablet_bootstrap.cc:654] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.677559 11134 raft_consensus.cc:357] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.677664 11134 raft_consensus.cc:738] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.677767 11134 consensus_queue.cc:260] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.677632 11135 raft_consensus.cc:357] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.677874 11135 raft_consensus.cc:738] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.678051 11134 ts_tablet_manager.cc:1428] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.678231 11134 tablet_bootstrap.cc:492] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.678205 11135 consensus_queue.cc:260] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.678718 11137 tablet_bootstrap.cc:492] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.678823 11137 ts_tablet_manager.cc:1397] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.679190 11135 ts_tablet_manager.cc:1428] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250114 03:05:44.679224 11134 tablet_bootstrap.cc:654] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.679414 11137 raft_consensus.cc:357] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.679526 11137 raft_consensus.cc:738] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.679605 10850 heartbeater.cc:502] Master 127.1.109.126:40153 was elected leader, sending a full tablet report...
I20250114 03:05:44.679637 11137 consensus_queue.cc:260] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.680045 11137 ts_tablet_manager.cc:1428] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250114 03:05:44.680248 11137 tablet_bootstrap.cc:492] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.680477 11134 tablet_bootstrap.cc:492] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.680555 11134 ts_tablet_manager.cc:1397] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.681051 11134 raft_consensus.cc:357] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.681149 11134 raft_consensus.cc:738] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.681250 11134 consensus_queue.cc:260] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.681342 11137 tablet_bootstrap.cc:654] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.681561 11134 ts_tablet_manager.cc:1428] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.681725 11134 tablet_bootstrap.cc:492] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.682689 11137 tablet_bootstrap.cc:492] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.682797 11134 tablet_bootstrap.cc:654] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.683802 11134 tablet_bootstrap.cc:492] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.683885 11134 ts_tablet_manager.cc:1397] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.684085 11135 tablet_bootstrap.cc:492] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.684084 11137 ts_tablet_manager.cc:1397] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250114 03:05:44.684334 11134 raft_consensus.cc:357] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.684437 11134 raft_consensus.cc:738] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.684538 11134 consensus_queue.cc:260] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.684816 11134 ts_tablet_manager.cc:1428] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.684984 11134 tablet_bootstrap.cc:492] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.685110 11137 raft_consensus.cc:357] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.685279 11135 tablet_bootstrap.cc:654] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.685312 11137 raft_consensus.cc:738] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.685542 11137 consensus_queue.cc:260] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.685879 11137 ts_tablet_manager.cc:1428] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250114 03:05:44.686092 11137 tablet_bootstrap.cc:492] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.686110 11134 tablet_bootstrap.cc:654] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.686811 11135 tablet_bootstrap.cc:492] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.686911 11135 ts_tablet_manager.cc:1397] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.687167 11134 tablet_bootstrap.cc:492] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.687248 11134 ts_tablet_manager.cc:1397] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.687601 11135 raft_consensus.cc:357] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.687651 11134 raft_consensus.cc:357] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.687736 11134 raft_consensus.cc:738] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.687743 11135 raft_consensus.cc:738] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.687861 11137 tablet_bootstrap.cc:654] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.687842 11134 consensus_queue.cc:260] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.687868 11135 consensus_queue.cc:260] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.688083 11134 ts_tablet_manager.cc:1428] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.688107 11135 ts_tablet_manager.cc:1428] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:44.688251 11135 tablet_bootstrap.cc:492] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.688251 11134 tablet_bootstrap.cc:492] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.689256 11134 tablet_bootstrap.cc:654] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.689308 11135 tablet_bootstrap.cc:654] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.690646 11135 tablet_bootstrap.cc:492] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.690725 11135 ts_tablet_manager.cc:1397] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20250114 03:05:44.690977 11134 tablet_bootstrap.cc:492] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.691056 11134 ts_tablet_manager.cc:1397] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.691184 11135 raft_consensus.cc:357] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.691300 11135 raft_consensus.cc:738] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.691391 11135 consensus_queue.cc:260] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.691496 11134 raft_consensus.cc:357] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.691604 11134 raft_consensus.cc:738] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.691701 11135 ts_tablet_manager.cc:1428] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250114 03:05:44.691709 11134 consensus_queue.cc:260] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.691888 11135 tablet_bootstrap.cc:492] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.692011 11134 ts_tablet_manager.cc:1428] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.692173 11134 tablet_bootstrap.cc:492] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.693034 11135 tablet_bootstrap.cc:654] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.693176 11134 tablet_bootstrap.cc:654] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.694404 11134 tablet_bootstrap.cc:492] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.694464 11135 tablet_bootstrap.cc:492] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.694494 11134 ts_tablet_manager.cc:1397] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.694530 11135 ts_tablet_manager.cc:1397] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.694948 11134 raft_consensus.cc:357] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.694948 11135 raft_consensus.cc:357] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.695060 11134 raft_consensus.cc:738] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.695060 11135 raft_consensus.cc:738] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.695158 11134 consensus_queue.cc:260] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.695169 11135 consensus_queue.cc:260] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.695418 11134 ts_tablet_manager.cc:1428] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.695448 11135 ts_tablet_manager.cc:1428] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.695590 11135 tablet_bootstrap.cc:492] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.695598 11134 tablet_bootstrap.cc:492] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8: Bootstrap starting.
I20250114 03:05:44.696668 11134 tablet_bootstrap.cc:654] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.696668 11135 tablet_bootstrap.cc:654] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.697428 11137 tablet_bootstrap.cc:492] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.697547 11137 ts_tablet_manager.cc:1397] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250114 03:05:44.697924 11134 tablet_bootstrap.cc:492] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8: No bootstrap required, opened a new log
I20250114 03:05:44.697958 11135 tablet_bootstrap.cc:492] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.697997 11134 ts_tablet_manager.cc:1397] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20250114 03:05:44.698024 11135 ts_tablet_manager.cc:1397] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20250114 03:05:44.698033 11137 raft_consensus.cc:357] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.698143 11137 raft_consensus.cc:738] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.698241 11137 consensus_queue.cc:260] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.698421 11135 raft_consensus.cc:357] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.698421 11134 raft_consensus.cc:357] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.698534 11137 ts_tablet_manager.cc:1428] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:44.698531 11134 raft_consensus.cc:738] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f37fe03b35f4b00a9a7cc9e315383e8, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.698531 11135 raft_consensus.cc:738] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.698621 11134 consensus_queue.cc:260] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.698627 11135 consensus_queue.cc:260] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.698885 11135 ts_tablet_manager.cc:1428] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250114 03:05:44.698910 11134 ts_tablet_manager.cc:1428] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.699047 11135 tablet_bootstrap.cc:492] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.699154 11137 tablet_bootstrap.cc:492] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.700047 11135 tablet_bootstrap.cc:654] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.700192 11137 tablet_bootstrap.cc:654] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.703325 11137 tablet_bootstrap.cc:492] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.703408 11137 ts_tablet_manager.cc:1397] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20250114 03:05:44.703707 11135 tablet_bootstrap.cc:492] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.703796 11135 ts_tablet_manager.cc:1397] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20250114 03:05:44.703821 11137 raft_consensus.cc:357] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.703949 11137 raft_consensus.cc:738] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.704077 11137 consensus_queue.cc:260] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.704295 11135 raft_consensus.cc:357] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.704373 11137 ts_tablet_manager.cc:1428] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:44.704407 11135 raft_consensus.cc:738] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.704521 11135 consensus_queue.cc:260] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.704557 11137 tablet_bootstrap.cc:492] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.704771 11135 ts_tablet_manager.cc:1428] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.704970 11135 tablet_bootstrap.cc:492] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.705643 11137 tablet_bootstrap.cc:654] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.706422 11135 tablet_bootstrap.cc:654] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.707803 11137 tablet_bootstrap.cc:492] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.707890 11137 ts_tablet_manager.cc:1397] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.708236 11135 tablet_bootstrap.cc:492] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.708375 11135 ts_tablet_manager.cc:1397] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20250114 03:05:44.708566 11137 raft_consensus.cc:357] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.708698 11137 raft_consensus.cc:738] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.708804 11137 consensus_queue.cc:260] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.709045 11135 raft_consensus.cc:357] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.709120 11137 ts_tablet_manager.cc:1428] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.709193 11135 raft_consensus.cc:738] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.709273 11137 tablet_bootstrap.cc:492] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.709353 11135 consensus_queue.cc:260] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.709636 11135 ts_tablet_manager.cc:1428] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.709830 11135 tablet_bootstrap.cc:492] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.710314 11137 tablet_bootstrap.cc:654] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.710830 11135 tablet_bootstrap.cc:654] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.712204 11137 tablet_bootstrap.cc:492] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.712294 11137 ts_tablet_manager.cc:1397] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.712486 11135 tablet_bootstrap.cc:492] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.712567 11135 ts_tablet_manager.cc:1397] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.712798 11137 raft_consensus.cc:357] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.712932 11137 raft_consensus.cc:738] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.713024 11137 consensus_queue.cc:260] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.713055 11135 raft_consensus.cc:357] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.713145 11135 raft_consensus.cc:738] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.713233 11135 consensus_queue.cc:260] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.713505 11135 ts_tablet_manager.cc:1428] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.713652 11135 tablet_bootstrap.cc:492] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.713665 11137 ts_tablet_manager.cc:1428] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.713809 11137 tablet_bootstrap.cc:492] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.714709 11135 tablet_bootstrap.cc:654] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.714759 11137 tablet_bootstrap.cc:654] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.716396 11137 tablet_bootstrap.cc:492] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.716487 11137 ts_tablet_manager.cc:1397] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20250114 03:05:44.716521 11135 tablet_bootstrap.cc:492] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.716588 11135 ts_tablet_manager.cc:1397] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.717026 11135 raft_consensus.cc:357] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.717026 11137 raft_consensus.cc:357] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.717131 11137 raft_consensus.cc:738] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.717131 11135 raft_consensus.cc:738] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.717226 11135 consensus_queue.cc:260] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.717242 11137 consensus_queue.cc:260] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.717499 11135 ts_tablet_manager.cc:1428] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.717564 11137 ts_tablet_manager.cc:1428] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.717679 11135 tablet_bootstrap.cc:492] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768: Bootstrap starting.
I20250114 03:05:44.717697 11137 tablet_bootstrap.cc:492] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.718629 11137 tablet_bootstrap.cc:654] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.718936 11135 tablet_bootstrap.cc:654] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.720103 11135 tablet_bootstrap.cc:492] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768: No bootstrap required, opened a new log
I20250114 03:05:44.720181 11135 ts_tablet_manager.cc:1397] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.720593 11137 tablet_bootstrap.cc:492] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.720579 11135 raft_consensus.cc:357] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.720654 11135 raft_consensus.cc:738] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6c11cd2ea1445e9764341bc17f3768, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.720675 11137 ts_tablet_manager.cc:1397] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.720731 11135 consensus_queue.cc:260] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.721050 11135 ts_tablet_manager.cc:1428] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20250114 03:05:44.721238 11137 raft_consensus.cc:357] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.721328 11137 raft_consensus.cc:738] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.721426 11137 consensus_queue.cc:260] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.721709 11137 ts_tablet_manager.cc:1428] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250114 03:05:44.721935 11137 tablet_bootstrap.cc:492] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Bootstrap starting.
I20250114 03:05:44.722986 11137 tablet_bootstrap.cc:654] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Neither blocks nor log segments found. Creating new log.
I20250114 03:05:44.724889 11137 tablet_bootstrap.cc:492] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: No bootstrap required, opened a new log
I20250114 03:05:44.724961 11137 ts_tablet_manager.cc:1397] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20250114 03:05:44.725327 11137 raft_consensus.cc:357] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.725411 11137 raft_consensus.cc:738] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Initialized, Role: FOLLOWER
I20250114 03:05:44.725482 11137 consensus_queue.cc:260] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.725723 11137 ts_tablet_manager.cc:1428] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250114 03:05:44.760406 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a9176121f8f94d32812a73e49717ae7a"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.760682 11066 raft_consensus.cc:491] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.760795 11066 raft_consensus.cc:3054] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.761816 11066 raft_consensus.cc:513] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.762827 11066 leader_election.cc:290] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.766932 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9176121f8f94d32812a73e49717ae7a" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.767151 10804 raft_consensus.cc:3054] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.767299 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9176121f8f94d32812a73e49717ae7a" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.767479 10935 raft_consensus.cc:3054] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.768448 10804 raft_consensus.cc:2463] T a9176121f8f94d32812a73e49717ae7a P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.768594 10935 raft_consensus.cc:2463] T a9176121f8f94d32812a73e49717ae7a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.769026 11001 leader_election.cc:304] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5041e64b77742878812e2e2d46ebf7b, cb6c11cd2ea1445e9764341bc17f3768; no voters:
I20250114 03:05:44.769320 11148 raft_consensus.cc:2798] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.769420 11148 raft_consensus.cc:695] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.769742 11148 consensus_queue.cc:237] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.772888 10661 catalog_manager.cc:5526] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.779786 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa5e3b41dc43497e9c13c9b0ee2e6aa0"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.779938 11066 raft_consensus.cc:491] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.780006 11066 raft_consensus.cc:3054] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.780862 11066 raft_consensus.cc:513] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.781221 11066 leader_election.cc:290] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.781643 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5e3b41dc43497e9c13c9b0ee2e6aa0" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.781771 10804 raft_consensus.cc:3054] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.781713 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa5e3b41dc43497e9c13c9b0ee2e6aa0" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.781852 10935 raft_consensus.cc:3054] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.782641 10804 raft_consensus.cc:2463] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.782699 10935 raft_consensus.cc:2463] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.782943 11001 leader_election.cc:304] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5041e64b77742878812e2e2d46ebf7b, cb6c11cd2ea1445e9764341bc17f3768; no voters:
I20250114 03:05:44.783118 11148 raft_consensus.cc:2798] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.783242 11148 raft_consensus.cc:695] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.783397 11148 consensus_queue.cc:237] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.785315 10661 catalog_manager.cc:5526] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.790357 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9fa771d443694da5bb7f11ae022093b4"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.790483 11066 raft_consensus.cc:491] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.790549 11066 raft_consensus.cc:3054] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.791316 11066 raft_consensus.cc:513] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.791651 11066 leader_election.cc:290] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.792055 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa771d443694da5bb7f11ae022093b4" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.792079 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fa771d443694da5bb7f11ae022093b4" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.792204 10804 raft_consensus.cc:3054] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.792214 10935 raft_consensus.cc:3054] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.793143 10935 raft_consensus.cc:2463] T 9fa771d443694da5bb7f11ae022093b4 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.793179 10804 raft_consensus.cc:2463] T 9fa771d443694da5bb7f11ae022093b4 P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.793465 10999 leader_election.cc:304] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f37fe03b35f4b00a9a7cc9e315383e8, a5041e64b77742878812e2e2d46ebf7b; no voters:
I20250114 03:05:44.793643 11148 raft_consensus.cc:2798] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.793731 11148 raft_consensus.cc:695] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.793874 11148 consensus_queue.cc:237] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.795557 10661 catalog_manager.cc:5526] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.800355 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39a4da5ec6e7480983dad214dff36e0d"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.800470 11066 raft_consensus.cc:491] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.800539 11066 raft_consensus.cc:3054] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.801337 11066 raft_consensus.cc:513] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.801676 11066 leader_election.cc:290] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.802040 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a4da5ec6e7480983dad214dff36e0d" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.802069 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39a4da5ec6e7480983dad214dff36e0d" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.802158 10804 raft_consensus.cc:3054] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.802174 10935 raft_consensus.cc:3054] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.803004 10935 raft_consensus.cc:2463] T 39a4da5ec6e7480983dad214dff36e0d P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.803130 10804 raft_consensus.cc:2463] T 39a4da5ec6e7480983dad214dff36e0d P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.803274 10999 leader_election.cc:304] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f37fe03b35f4b00a9a7cc9e315383e8, a5041e64b77742878812e2e2d46ebf7b; no voters:
I20250114 03:05:44.803452 11148 raft_consensus.cc:2798] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.803542 11148 raft_consensus.cc:695] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.803664 11148 consensus_queue.cc:237] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.805544 10661 catalog_manager.cc:5526] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.810932 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f3350d31a6834e808c03cdfabd86ed5a"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.811079 11066 raft_consensus.cc:491] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.811141 11066 raft_consensus.cc:3054] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.811915 11066 raft_consensus.cc:513] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.812274 11066 leader_election.cc:290] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.812674 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3350d31a6834e808c03cdfabd86ed5a" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.812734 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3350d31a6834e808c03cdfabd86ed5a" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.812816 10804 raft_consensus.cc:3054] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.812927 10935 raft_consensus.cc:3054] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.813740 10804 raft_consensus.cc:2463] T f3350d31a6834e808c03cdfabd86ed5a P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.813784 10935 raft_consensus.cc:2463] T f3350d31a6834e808c03cdfabd86ed5a P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.814045 11001 leader_election.cc:304] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5041e64b77742878812e2e2d46ebf7b, cb6c11cd2ea1445e9764341bc17f3768; no voters:
I20250114 03:05:44.814254 11143 raft_consensus.cc:2798] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.814365 11143 raft_consensus.cc:695] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.814523 11143 consensus_queue.cc:237] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.816285 10661 catalog_manager.cc:5526] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.821003 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9837cc4d27ce464e910753a7186e3405"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.821107 11066 raft_consensus.cc:491] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.821159 11066 raft_consensus.cc:3054] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.821856 11066 raft_consensus.cc:513] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.822160 11066 leader_election.cc:290] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.822538 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9837cc4d27ce464e910753a7186e3405" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.822676 10935 raft_consensus.cc:3054] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.822674 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9837cc4d27ce464e910753a7186e3405" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.822790 10804 raft_consensus.cc:3054] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.823633 10804 raft_consensus.cc:2463] T 9837cc4d27ce464e910753a7186e3405 P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.823634 10935 raft_consensus.cc:2463] T 9837cc4d27ce464e910753a7186e3405 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.823976 10999 leader_election.cc:304] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f37fe03b35f4b00a9a7cc9e315383e8, a5041e64b77742878812e2e2d46ebf7b; no voters:
I20250114 03:05:44.824193 11143 raft_consensus.cc:2798] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.824270 11143 raft_consensus.cc:695] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.824409 11143 consensus_queue.cc:237] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.826267 10661 catalog_manager.cc:5526] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
W20250114 03:05:44.829001 11113 tablet.cc:2367] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:44.831353 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7ba004ea79944fe4bd8cf4e31bbc07d7"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.831473 11066 raft_consensus.cc:491] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.831521 11066 raft_consensus.cc:3054] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.832226 11066 raft_consensus.cc:513] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.832531 11066 leader_election.cc:290] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.832947 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba004ea79944fe4bd8cf4e31bbc07d7" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.832960 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba004ea79944fe4bd8cf4e31bbc07d7" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.833082 10935 raft_consensus.cc:3054] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.833099 10804 raft_consensus.cc:3054] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.833962 10935 raft_consensus.cc:2463] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.833979 10804 raft_consensus.cc:2463] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.834216 10999 leader_election.cc:304] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f37fe03b35f4b00a9a7cc9e315383e8, a5041e64b77742878812e2e2d46ebf7b; no voters:
I20250114 03:05:44.834482 11143 raft_consensus.cc:2798] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.834555 11143 raft_consensus.cc:695] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.834697 11143 consensus_queue.cc:237] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.836421 10661 catalog_manager.cc:5526] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.841252 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "26040bfbffe047818ba6fb4bb22208c5"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.841382 11066 raft_consensus.cc:491] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.841444 11066 raft_consensus.cc:3054] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.842187 11066 raft_consensus.cc:513] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.842546 11066 leader_election.cc:290] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.842978 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26040bfbffe047818ba6fb4bb22208c5" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.843065 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26040bfbffe047818ba6fb4bb22208c5" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.843135 10935 raft_consensus.cc:3054] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.843216 10804 raft_consensus.cc:3054] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.844113 10935 raft_consensus.cc:2463] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.844208 10804 raft_consensus.cc:2463] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.844431 10999 leader_election.cc:304] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f37fe03b35f4b00a9a7cc9e315383e8, a5041e64b77742878812e2e2d46ebf7b; no voters:
I20250114 03:05:44.844645 11143 raft_consensus.cc:2798] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.844755 11143 raft_consensus.cc:695] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.844924 11143 consensus_queue.cc:237] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.846782 10661 catalog_manager.cc:5526] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
I20250114 03:05:44.851496 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e504fd6dfe5c4670a976d1e6e5bd2e01"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.851626 11066 raft_consensus.cc:491] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.851666 11066 raft_consensus.cc:3054] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.852363 11066 raft_consensus.cc:513] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.852703 11066 leader_election.cc:290] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.853142 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e504fd6dfe5c4670a976d1e6e5bd2e01" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.853159 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e504fd6dfe5c4670a976d1e6e5bd2e01" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.853287 10935 raft_consensus.cc:3054] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.853289 10804 raft_consensus.cc:3054] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.854171 10804 raft_consensus.cc:2463] T e504fd6dfe5c4670a976d1e6e5bd2e01 P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.854175 10935 raft_consensus.cc:2463] T e504fd6dfe5c4670a976d1e6e5bd2e01 P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.854492 11001 leader_election.cc:304] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5041e64b77742878812e2e2d46ebf7b, cb6c11cd2ea1445e9764341bc17f3768; no voters:
I20250114 03:05:44.854704 11143 raft_consensus.cc:2798] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.854797 11143 raft_consensus.cc:695] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.854934 11143 consensus_queue.cc:237] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.856791 10661 catalog_manager.cc:5526] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } }
W20250114 03:05:44.857473 10982 tablet.cc:2367] T 26040bfbffe047818ba6fb4bb22208c5 P 9f37fe03b35f4b00a9a7cc9e315383e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:44.862190 11066 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6537b58b18c141caaa7935e56110a5dd"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.862344 11066 raft_consensus.cc:491] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250114 03:05:44.862394 11066 raft_consensus.cc:3054] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.863145 11066 raft_consensus.cc:513] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.863515 11066 leader_election.cc:290] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Requested vote from peers cb6c11cd2ea1445e9764341bc17f3768 (127.1.109.65:41359), 9f37fe03b35f4b00a9a7cc9e315383e8 (127.1.109.66:35371)
I20250114 03:05:44.864025 10804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6537b58b18c141caaa7935e56110a5dd" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb6c11cd2ea1445e9764341bc17f3768"
I20250114 03:05:44.864048 10935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6537b58b18c141caaa7935e56110a5dd" candidate_uuid: "a5041e64b77742878812e2e2d46ebf7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8"
I20250114 03:05:44.864187 10804 raft_consensus.cc:3054] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.864187 10935 raft_consensus.cc:3054] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 0 FOLLOWER]: Advancing to term 1
I20250114 03:05:44.865201 10804 raft_consensus.cc:2463] T 6537b58b18c141caaa7935e56110a5dd P cb6c11cd2ea1445e9764341bc17f3768 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.865201 10935 raft_consensus.cc:2463] T 6537b58b18c141caaa7935e56110a5dd P 9f37fe03b35f4b00a9a7cc9e315383e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5041e64b77742878812e2e2d46ebf7b in term 1.
I20250114 03:05:44.865514 10999 leader_election.cc:304] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f37fe03b35f4b00a9a7cc9e315383e8, a5041e64b77742878812e2e2d46ebf7b; no voters:
I20250114 03:05:44.865692 11143 raft_consensus.cc:2798] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Leader election won for term 1
I20250114 03:05:44.865793 11143 raft_consensus.cc:695] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Leader. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.865945 11143 consensus_queue.cc:237] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.867837 10661 catalog_manager.cc:5526] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5041e64b77742878812e2e2d46ebf7b (127.1.109.67). New cstate: current_term: 1 leader_uuid: "a5041e64b77742878812e2e2d46ebf7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } health_report { overall_health: UNKNOWN } } }
I20250114 03:05:44.873220 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet a9176121f8f94d32812a73e49717ae7a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.873718 11184 tablet_replica.cc:331] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.874023 11184 raft_consensus.cc:2238] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:44.874363 11184 pending_rounds.cc:62] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.874459 11184 pending_rounds.cc:69] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030877779570688 op_type: NO_OP noop_request { }
I20250114 03:05:44.874568 11184 raft_consensus.cc:2883] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.874651 11184 raft_consensus.cc:2267] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.875839 11184 ts_tablet_manager.cc:1905] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.877907 11184 ts_tablet_manager.cc:1918] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.877993 11184 log.cc:1198] T a9176121f8f94d32812a73e49717ae7a P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/a9176121f8f94d32812a73e49717ae7a
I20250114 03:05:44.883994 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet fa5e3b41dc43497e9c13c9b0ee2e6aa0 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.884245 11184 tablet_replica.cc:331] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.884382 11184 raft_consensus.cc:2238] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:44.884574 11184 pending_rounds.cc:62] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.884606 11184 pending_rounds.cc:69] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030877834313728 op_type: NO_OP noop_request { }
I20250114 03:05:44.884682 11184 raft_consensus.cc:2883] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.884744 11184 raft_consensus.cc:2267] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.885337 11184 ts_tablet_manager.cc:1905] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.887024 11184 ts_tablet_manager.cc:1918] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.887096 11184 log.cc:1198] T fa5e3b41dc43497e9c13c9b0ee2e6aa0 P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/fa5e3b41dc43497e9c13c9b0ee2e6aa0
I20250114 03:05:44.893224 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fa771d443694da5bb7f11ae022093b4 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.893455 11184 tablet_replica.cc:331] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.893599 11184 raft_consensus.cc:2238] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:44.893764 11184 pending_rounds.cc:62] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.893796 11184 pending_rounds.cc:69] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030877877071872 op_type: NO_OP noop_request { }
I20250114 03:05:44.893877 11184 raft_consensus.cc:2883] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.893936 11184 raft_consensus.cc:2267] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.894558 11184 ts_tablet_manager.cc:1905] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.896138 11184 ts_tablet_manager.cc:1918] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.896204 11184 log.cc:1198] T 9fa771d443694da5bb7f11ae022093b4 P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/9fa771d443694da5bb7f11ae022093b4
I20250114 03:05:44.901770 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 39a4da5ec6e7480983dad214dff36e0d with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.901953 11184 tablet_replica.cc:331] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.902083 11184 raft_consensus.cc:2238] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:44.902240 11184 pending_rounds.cc:62] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.902273 11184 pending_rounds.cc:69] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030877916995584 op_type: NO_OP noop_request { }
I20250114 03:05:44.902347 11184 raft_consensus.cc:2883] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.902408 11184 raft_consensus.cc:2267] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.903025 11184 ts_tablet_manager.cc:1905] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.904616 11184 ts_tablet_manager.cc:1918] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.904682 11184 log.cc:1198] T 39a4da5ec6e7480983dad214dff36e0d P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/39a4da5ec6e7480983dad214dff36e0d
I20250114 03:05:44.910305 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet f3350d31a6834e808c03cdfabd86ed5a with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.910490 11184 tablet_replica.cc:331] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.910625 11184 raft_consensus.cc:2238] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Raft consensus shutting down.
I20250114 03:05:44.910782 11184 pending_rounds.cc:62] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.910815 11184 pending_rounds.cc:69] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030877961838592 op_type: NO_OP noop_request { }
I20250114 03:05:44.910887 11184 raft_consensus.cc:2883] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.910948 11184 raft_consensus.cc:2267] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.911559 11184 ts_tablet_manager.cc:1905] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.913173 11184 ts_tablet_manager.cc:1918] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.913240 11184 log.cc:1198] T f3350d31a6834e808c03cdfabd86ed5a P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/f3350d31a6834e808c03cdfabd86ed5a
I20250114 03:05:44.918962 11066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9837cc4d27ce464e910753a7186e3405"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.919123 11066 raft_consensus.cc:589] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Received request to step down
I20250114 03:05:44.919173 11066 raft_consensus.cc:3049] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:44.919205 11066 raft_consensus.cc:738] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.919348 11066 consensus_queue.cc:260] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.919477 11066 raft_consensus.cc:3054] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:44.924866 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 9837cc4d27ce464e910753a7186e3405 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.925100 11184 tablet_replica.cc:331] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.925226 11184 raft_consensus.cc:2238] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:44.925318 11184 pending_rounds.cc:62] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.925340 11184 pending_rounds.cc:69] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030878002458624 op_type: NO_OP noop_request { }
I20250114 03:05:44.925397 11184 raft_consensus.cc:2883] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.925431 11184 raft_consensus.cc:2267] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus is shut down!
W20250114 03:05:44.925571 10851 tablet.cc:2367] T 26040bfbffe047818ba6fb4bb22208c5 P cb6c11cd2ea1445e9764341bc17f3768: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250114 03:05:44.926059 11184 ts_tablet_manager.cc:1905] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.927846 11184 ts_tablet_manager.cc:1918] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.927907 11184 log.cc:1198] T 9837cc4d27ce464e910753a7186e3405 P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/9837cc4d27ce464e910753a7186e3405
I20250114 03:05:44.933389 11066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7ba004ea79944fe4bd8cf4e31bbc07d7"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.933558 11066 raft_consensus.cc:589] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Received request to step down
I20250114 03:05:44.933619 11066 raft_consensus.cc:3049] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:44.933655 11066 raft_consensus.cc:738] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.933784 11066 consensus_queue.cc:260] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.933907 11066 raft_consensus.cc:3054] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:44.941375 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba004ea79944fe4bd8cf4e31bbc07d7 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.941579 11184 tablet_replica.cc:331] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.941710 11184 raft_consensus.cc:2238] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:44.941814 11184 pending_rounds.cc:62] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.941850 11184 pending_rounds.cc:69] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030878044246016 op_type: NO_OP noop_request { }
I20250114 03:05:44.941918 11184 raft_consensus.cc:2883] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.941977 11184 raft_consensus.cc:2267] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.942647 11184 ts_tablet_manager.cc:1905] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.944447 11184 ts_tablet_manager.cc:1918] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.944530 11184 log.cc:1198] T 7ba004ea79944fe4bd8cf4e31bbc07d7 P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/7ba004ea79944fe4bd8cf4e31bbc07d7
I20250114 03:05:44.951550 11066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "26040bfbffe047818ba6fb4bb22208c5"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.951719 11066 raft_consensus.cc:589] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Received request to step down
I20250114 03:05:44.951781 11066 raft_consensus.cc:3049] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:44.951817 11066 raft_consensus.cc:738] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.951956 11066 consensus_queue.cc:260] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.952091 11066 raft_consensus.cc:3054] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:44.957856 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 26040bfbffe047818ba6fb4bb22208c5 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.958120 11184 tablet_replica.cc:331] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.958281 11184 raft_consensus.cc:2238] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:44.958401 11184 pending_rounds.cc:62] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.958441 11184 pending_rounds.cc:69] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030878086459392 op_type: NO_OP noop_request { }
I20250114 03:05:44.958529 11184 raft_consensus.cc:2883] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.958595 11184 raft_consensus.cc:2267] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.959286 11184 ts_tablet_manager.cc:1905] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.961081 11184 ts_tablet_manager.cc:1918] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.961151 11184 log.cc:1198] T 26040bfbffe047818ba6fb4bb22208c5 P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/26040bfbffe047818ba6fb4bb22208c5
I20250114 03:05:44.966904 11066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e504fd6dfe5c4670a976d1e6e5bd2e01"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.967068 11066 raft_consensus.cc:589] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Received request to step down
I20250114 03:05:44.967128 11066 raft_consensus.cc:3049] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:44.967150 11066 raft_consensus.cc:738] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.967278 11066 consensus_queue.cc:260] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } }
I20250114 03:05:44.967407 11066 raft_consensus.cc:3054] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:44.972863 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet e504fd6dfe5c4670a976d1e6e5bd2e01 with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.973098 11184 tablet_replica.cc:331] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.973245 11184 raft_consensus.cc:2238] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:44.973366 11184 pending_rounds.cc:62] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.973400 11184 pending_rounds.cc:69] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030878127247360 op_type: NO_OP noop_request { }
I20250114 03:05:44.973479 11184 raft_consensus.cc:2883] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.973538 11184 raft_consensus.cc:2267] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.974164 11184 ts_tablet_manager.cc:1905] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.975930 11184 ts_tablet_manager.cc:1918] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.976003 11184 log.cc:1198] T e504fd6dfe5c4670a976d1e6e5bd2e01 P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/e504fd6dfe5c4670a976d1e6e5bd2e01
I20250114 03:05:44.982066 11066 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6537b58b18c141caaa7935e56110a5dd"
dest_uuid: "a5041e64b77742878812e2e2d46ebf7b"
from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.982244 11066 raft_consensus.cc:589] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Received request to step down
I20250114 03:05:44.982303 11066 raft_consensus.cc:3049] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Stepping down as leader of term 1
I20250114 03:05:44.982348 11066 raft_consensus.cc:738] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5041e64b77742878812e2e2d46ebf7b, State: Running, Role: LEADER
I20250114 03:05:44.982501 11066 consensus_queue.cc:260] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cb6c11cd2ea1445e9764341bc17f3768" member_type: VOTER last_known_addr { host: "127.1.109.65" port: 41359 } } peers { permanent_uuid: "a5041e64b77742878812e2e2d46ebf7b" member_type: VOTER last_known_addr { host: "127.1.109.67" port: 35995 } } peers { permanent_uuid: "9f37fe03b35f4b00a9a7cc9e315383e8" member_type: VOTER last_known_addr { host: "127.1.109.66" port: 35371 } }
I20250114 03:05:44.982641 11066 raft_consensus.cc:3054] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 1 FOLLOWER]: Advancing to term 2
I20250114 03:05:44.988328 11042 tablet_service.cc:1514] Processing DeleteTablet for tablet 6537b58b18c141caaa7935e56110a5dd with delete_type TABLET_DATA_TOMBSTONED from {username='slave'} at 127.0.0.1:53814
I20250114 03:05:44.988590 11184 tablet_replica.cc:331] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: stopping tablet replica
I20250114 03:05:44.988739 11184 raft_consensus.cc:2238] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus shutting down.
I20250114 03:05:44.988884 11184 pending_rounds.cc:62] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Trying to abort 1 pending ops.
I20250114 03:05:44.988914 11184 pending_rounds.cc:69] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7114030878172463104 op_type: NO_OP noop_request { }
I20250114 03:05:44.988998 11184 raft_consensus.cc:2883] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250114 03:05:44.989058 11184 raft_consensus.cc:2267] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250114 03:05:44.989720 11184 ts_tablet_manager.cc:1905] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250114 03:05:44.991515 11184 ts_tablet_manager.cc:1918] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.1
I20250114 03:05:44.991590 11184 log.cc:1198] T 6537b58b18c141caaa7935e56110a5dd P a5041e64b77742878812e2e2d46ebf7b: Deleting WAL directory at /tmp/dist-test-taskyEDKFA/test-tmp/raft_consensus_election-itest.0.NumReplicas_RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric_1.1736823772370706-1461-0/raft_consensus-itest-cluster/ts-2/wal/wals/6537b58b18c141caaa7935e56110a5dd
I20250114 03:05:44.998339 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10722
I20250114 03:05:45.005184 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10853
I20250114 03:05:45.011986 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10984
I20250114 03:05:45.018595 1461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskyEDKFA/build/debug/bin/kudu with pid 10631
2025-01-14T03:05:45Z chronyd exiting
[ OK ] NumReplicas/RaftConsensusNumLeadersMetricTest.TestNumLeadersMetric/1 (1289 ms)
[----------] 2 tests from NumReplicas/RaftConsensusNumLeadersMetricTest (2076 ms total)
[----------] Global test environment tear-down
[==========] 14 tests from 2 test suites ran. (172664 ms total)
[ PASSED ] 13 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] RaftConsensusElectionITest.TestNewLeaderCantResolvePeers
1 FAILED TEST
I20250114 03:05:45.045264 1461 logging.cc:424] LogThrottler /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/client/meta_cache.cc:302: suppressed but not reported on 13 messages since previous log ~10 seconds ago